Three-dimensional brain midline segmentation method, device, equipment, storage medium and program product

Abstract

The application discloses a three-dimensional brain midline segmentation method, device, equipment, storage medium and program product, and belongs to the technical field of image processing. The method comprises the following steps: performing feature coding on a sample three-dimensional brain image through a feature encoder to obtain a sample coding feature map; performing lesion area prediction, brain midline segmentation prediction and ideal brain midface prediction based on the sample coding feature map to obtain a lesion area prediction result, a brain midline segmentation prediction result and an ideal brain midface prediction result; and training the feature encoder based on the lesion area prediction result, the brain midline segmentation prediction result and the ideal brain midface prediction result. The method can improve the feature extraction accuracy of the feature coding network for the related features in the brain midline segmentation prediction by additionally adding the lesion area prediction task and the ideal brain midface prediction task, so that the accuracy of the brain midline segmentation prediction task is further improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a three-dimensional brain midline segmentation method, device, equipment, storage medium and program product. BACKGROUND

[0002] In the medical field, the brain midline is an anatomical structure that separates the left and right hemispheres. How to quickly and accurately identify the brain midline plays an important role in clinical and scientific research fields.

[0003] In related technologies, the brain midline is usually identified based on a two-dimensional brain image. However, the brain midline identified in this way is a curve in a two-dimensional plane, and it is not possible to know the specific anatomical structure inside the brain based on only one curve, which is not conducive to determining the state of the user's brain. Therefore, how to accurately predict the three-dimensional brain midline plays an important role in subsequent diagnosis and treatment by doctors. SUMMARY

[0004] The present application provides a three-dimensional brain midline segmentation method, device, equipment, storage medium and program product. The technical solution is as follows:

[0005] According to one aspect of the present application, a three-dimensional brain midline segmentation method is provided, the method comprising:

[0006] encoding features of a sample three-dimensional brain image by a feature encoder to obtain a sample encoded feature map;

[0007] performing lesion region prediction, brain midline segmentation prediction and ideal brain midplane prediction based on the sample encoded feature map to obtain a lesion region prediction result, a brain midline segmentation prediction result and an ideal brain midplane prediction result;

[0008] training the feature encoder based on the lesion region prediction result, the brain midline segmentation prediction result and the ideal brain midplane prediction result.

[0009] According to another aspect of the present application, a three-dimensional brain midline segmentation device is provided, the device comprising:

[0010] a feature encoding module configured to encode features of a sample three-dimensional brain image by a feature encoder to obtain a sample encoded feature map;

[0011] a multi-task segmentation prediction module configured to perform lesion region prediction, brain midline segmentation prediction and ideal brain midplane prediction based on the sample encoded feature map to obtain a lesion region prediction result, a brain midline segmentation prediction result and an ideal brain midplane prediction result;

[0012] The training module is configured to jointly train the feature encoder based on the lesion area prediction result, the brain midline segmentation prediction result, and the ideal brain midplane prediction result.

[0013] According to another aspect of the present application, a computer device is provided, which comprises a processor and a memory. The memory stores a computer program, which is loaded and executed by the processor to implement the three-dimensional brain midline segmentation method according to the above aspect.

[0014] According to another aspect of the present application, a computer readable storage medium is provided, which stores a computer program. The computer program is loaded and executed by a processor to implement the three-dimensional brain midline segmentation method according to the above aspect.

[0015] According to another aspect of the present application, a computer program product is provided, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the three-dimensional brain midline segmentation method according to the above aspect.

[0016] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects:

[0017] A three-dimensional brain midline segmentation method is provided. The sample three-dimensional brain image is feature-encoded, and multi-task prediction (lesion area prediction, brain midline segmentation prediction, and ideal brain midplane prediction) is performed based on the feature encoding result (sample encoding feature map) respectively. Since the lesion area will cause lateral deviation of the brain midline structure, and the ideal brain midplane is close to the position of the three-dimensional brain midline, by additionally adding the lesion area prediction task and the ideal brain midplane prediction task, the feature extraction accuracy of the feature encoding network for the related features in the brain midline segmentation prediction can be improved, thereby further improving the accuracy of the brain midline segmentation prediction task. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical schemes in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0019] Figure 1 is a schematic diagram of a computer system according to an example embodiment of the present application;

[0020] Figure 2 shows a flowchart of the three-dimensional brain midline segmentation method according to an example embodiment of the present application;

[0021] Figure 3 A flow chart of a three-dimensional brain midline segmentation method provided by another example embodiment of the present application is shown;

[0022] Figure 4 A schematic diagram of a model training process shown by an example embodiment of the present application is shown;

[0023] Figure 5 A flow chart of a three-dimensional brain midline segmentation method provided by another example embodiment of the present application is shown;

[0024] Figure 6 A schematic diagram of a distance map and a weight map shown by an example embodiment of the present application is shown;

[0025] Figure 7 A schematic diagram of a model training process shown by another example embodiment of the present application is shown;

[0026] Figure 8 A flow chart of a three-dimensional brain midline segmentation method provided by another example embodiment of the present application is shown;

[0027] Figure 9 A process schematic diagram of model training shown by another example embodiment of the present application is shown;

[0028] Figure 10 A flow chart of a three-dimensional brain midline segmentation method provided by another example embodiment of the present application is shown;

[0029] Figure 11 A process schematic diagram of brain midline offset determination shown by an example embodiment of the present application is shown;

[0030] Figure 12 A structural block diagram of a three-dimensional brain midline segmentation device provided by an example embodiment of the present application is shown;

[0031] Figure 13 A structural schematic diagram of a computer device according to an example embodiment is shown. DETAILED DESCRIPTION

[0032] To make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0033] First, the terms involved in the embodiments of the present application are briefly introduced.

[0034] Brain midline: is the anatomical structure for dividing the left and right brain, which can also be called the actual brain midline. In a two-dimensional brain image, the brain midline is a curve, and in a three-dimensional brain image, the brain midline is a curved surface for dividing the left and right brain, which is the intersection region of the left and right brain.

[0035] Ideal mid-sagittal plane: anatomically ideal mid-sagittal plane for roughly dividing the brain into two symmetrical hemispheres, which is a plane in a three-dimensional brain image.

[0036] Three-dimensional midline shift: the maximum perpendicular distance between the actual midline (surface) and the ideal mid-sagittal plane (plane) in three-dimensional space. In a clinical scenario, the three-dimensional midline shift can reflect the degree of lateral displacement of the midline structure by intracranial space-occupying lesions such as brain hematoma, tumor, and abscess. It is an important reference for the increase of intracranial pressure, has important significance for measuring the development of the disease, and can be used together with other parameters to determine the urgency of neurosurgical intervention and predict the clinical outcome of patients with space-occupying lesions. Therefore, integrating a three-dimensional midline shift calculation module based on computed tomography (CT) images in medical image analysis software has an important role in providing reference information for doctors and assisting doctors in diagnosis.

[0037] In this application, how to segment the three-dimensional midline corresponding to the three-dimensional brain image will be emphasized. Based on the three-dimensional midline combined with the ideal mid-sagittal plane, the three-dimensional midline shift corresponding to the three-dimensional brain image is determined to provide a reference for the brain condition in a clinical scenario.

[0038] Figure 1 is a schematic diagram of a computer system shown in an exemplary embodiment of the present application. As shown in Figure 1 , the computer system includes a first device 110 and a second device 120.

[0039] The first device 110 is a training device for training a segmentation model for segmenting a three-dimensional midline. After the training of the segmentation model is completed, the first device 110 can send the trained segmentation model to the second device 120 for deployment of the segmentation model in the second device 120. The second device 120 is a device for segmenting a three-dimensional midline using the segmentation model.

[0040] The process of model training by the first device 110 is as follows: inputting a sample three-dimensional brain image 111 into a feature encoding network 112 to obtain a sample encoding feature map 113 output by the feature encoding network 112, performing multi-task prediction based on the sample encoding feature map 113: lesion area prediction, midline segmentation prediction, and ideal mid-sagittal plane prediction, and then training the feature encoding network 112 through multi-task joint training.

[0041] Optionally, the first device 110 and the second device 120 described above can be computer devices with machine learning capabilities, such as terminals or servers.

[0042] Optionally, the first device 110 and the second device 120 can be the same computer device, or the first device 110 and the second device 120 can also be different computer devices. Moreover, when the first device 110 and the second device 120 are different devices, the first device 110 and the second device 120 can be the same type of device, such as the first device 110 and the second device 120 can both be servers; or the first device 110 and the second device 120 can also be different types of devices. The server can be a standalone physical server, or a server cluster or distributed system formed by multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal can be a smartphone, a vehicle-mounted terminal, a smart television, a wearable device, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, and the like, but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the present application.

[0043] Please refer to Figure 2 which shows a flowchart of a three-dimensional brain midline segmentation method provided by an example embodiment of the present application. The method is applied to Figure 1 The method is exemplarily described by taking the first device 110 shown in the figure as an example, and the method comprises the following steps.

[0044] In step 201, a feature encoder is used to perform feature encoding on a sample three-dimensional brain image, to obtain a sample encoded feature map.

[0045] Since the three-dimensional brain midline is a curved surface in the three-dimensional structure of the brain, in order to extract the three-dimensional brain midline, a plurality of sample three-dimensional brain images need to be prepared in advance before model training, so that the three-dimensional brain midline region features can be learned from the sample three-dimensional brain images in the subsequent model training process, and then the sample three-dimensional brain midline can be segmented from the sample three-dimensional brain images.

[0046] The three-dimensional brain midline segmentation model can include two parts: a feature encoder and a feature decoder. The feature encoder is used for feature extraction, and the feature decoder is used for feature reconstruction. In a possible implementation, the sample three-dimensional brain image is first input into the feature encoder, and the feature encoder performs feature encoding (feature extraction) based on the sample three-dimensional brain image, to obtain a sample encoded feature map, so that the feature encoder can perform feature reconstruction based on the sample encoded feature map.

[0047] Optionally, the manner of obtaining the sample three-dimensional brain image by the first device is as follows: the first device can obtain a plurality of continuous two-dimensional brain images, reconstruct the plurality of continuous two-dimensional brain images into a three-dimensional brain image, and determine the three-dimensional brain image as the sample three-dimensional brain image. The two-dimensional brain image can be a CT image of the brain, a Magnetic Resonance Imaging (MRI) image of the brain, or another type of brain scan image.

[0048] Optionally, the feature encoding network can use a 3D Unet network for feature extraction; the feature encoding network outputs a 96-channel sample encoding feature map, and each branch subsequently decodes the features.

[0049] At step 202, the lesion region prediction result, the brain midline segmentation prediction result, and the ideal brain mid-surface prediction result are obtained based on the sample encoding feature map.

[0050] To improve the segmentation prediction effect of the model on the three-dimensional brain midline, a plurality of specific prediction tasks are introduced in the model training process to jointly train the segmentation model. In one possible implementation, in addition to the brain midline segmentation task, a lesion region prediction task and an ideal brain mid-surface prediction task are also set to jointly train the segmentation model based on the three specific prediction tasks. After the feature encoding network outputs the sample encoding feature map, the lesion region prediction, the brain midline segmentation prediction, and the ideal brain mid-surface prediction are performed based on the same sample encoding feature map to obtain the lesion region prediction result, the brain midline segmentation prediction result, and the ideal brain mid-surface prediction result, so that the feature encoder can be trained based on the prediction results of the three tasks.

[0051] The brain midline segmentation prediction result is a prediction result of the three-dimensional brain midline in the three-dimensional brain image. In this embodiment, the brain midline segmentation prediction task is converted into a left and right brain segmentation task, and the brain midline segmentation prediction result is a prediction result of the left brain region, a prediction result of the right brain region, and a prediction result of the background region in the three-dimensional brain image. The three-dimensional brain midline in the three-dimensional brain image can be determined based on the prediction result of the left brain region and the prediction result of the right brain region.

[0052] Optionally, the lesion prediction result refers to a prediction result of the lesion region in the three-dimensional brain image, and the ideal brain mid-surface prediction result refers to a prediction result of the ideal brain mid-surface in the three-dimensional brain image.

[0053] Since the (actual) three-dimensional brain midline is a structure for segmenting the left and right hemispheres in a three-dimensional brain image, the three-dimensional brain midline segmentation model (segmentation model) needs to pay more attention to the intersection region of the left and right hemispheres in the three-dimensional brain image during the three-dimensional brain midline segmentation prediction task. Therefore, in order to improve the accuracy of feature extraction of the intersection region of the left and right hemispheres by the feature encoding network, a lesion region prediction task and an ideal brain midplane prediction task are additionally set. The lesion region prediction task is used to predict the location of the lesion region in the three-dimensional brain image. Since the lesion region in the brain will cause lateral deviation of the brain midline structure, accurately predicting the lesion region is helpful for predicting the location of the three-dimensional brain midline in the three-dimensional brain image. The ideal brain midplane prediction task is used to predict the location of the ideal brain midplane in the three-dimensional brain image. Since the ideal brain midplane of the brain is a plane result for segmenting the brain into left and right hemispheres, which is close to the location of the three-dimensional brain midline, accurately predicting the ideal brain midplane is also helpful for predicting the location of the three-dimensional brain midline in the three-dimensional brain image.

[0054] In step 203, the feature encoder is jointly trained based on the lesion region prediction result, the brain midline segmentation prediction result and the ideal brain midplane prediction result.

[0055] In a possible implementation, after obtaining the lesion region prediction result, the brain midline segmentation prediction result and the ideal brain midplane prediction result, the feature encoder can be jointly trained based on the lesion region prediction result, the brain midline segmentation prediction result and the ideal brain midplane prediction result, so that the feature encoder extracts more features related to the brain midline segmentation prediction, thereby further improving the segmentation accuracy of the three-dimensional brain midline.

[0056] In a possible implementation, after obtaining the lesion region prediction result, the brain midline segmentation prediction result and the ideal brain midplane prediction result, the feature encoder can be jointly trained based on the lesion region prediction result, the brain midline segmentation prediction result and the ideal brain midplane prediction result, so that the feature encoder extracts more features related to the brain midline segmentation prediction, thereby further improving the segmentation accuracy of the three-dimensional brain midline.

[0057] In summary, the three-dimensional brain midline segmentation method provided in the embodiments of the present application is as follows: feature encoding is performed on a sample three-dimensional brain image, and multi-task prediction (lesion region prediction, three-dimensional brain midline segmentation prediction, and ideal brain midline surface prediction) is performed based on the feature encoding result (sample encoding feature map). Since the lesion region will cause lateral deviation of the brain midline structure, and the ideal brain midline surface is close to the position of the three-dimensional brain midline, by additionally adding the lesion region prediction task and the ideal brain midline surface prediction task, the feature extraction accuracy of the feature encoding network for the relevant features in the three-dimensional brain midline segmentation prediction can be improved, thereby further improving the accuracy of the three-dimensional brain midline segmentation prediction task.

[0058] In the embodiments, the sample encoding feature map output by the feature encoder can be shared by multiple prediction tasks, and the sample encoding feature map is reconstructed by different feature decoders to obtain the prediction results corresponding to different prediction tasks, respectively.

[0059] Reference is made to Figure 3 which shows a flowchart of a three-dimensional brain midline segmentation method provided by another exemplary embodiment of the present application. The method is applied to Figure 1 and is exemplified by the first device 110 shown in the figure. The method comprises the following steps:

[0060] In step 301, feature encoding is performed on the sample three-dimensional brain image by a feature encoder to obtain a sample encoding feature map.

[0061] The implementation of step 301 can refer to step 201, and the embodiments will not be described here.

[0062] In step 302, lesion region prediction is performed based on the sample encoding feature map to obtain a first sample probability map. The first sample probability map is used to represent the probability of each pixel point in the sample three-dimensional brain image belonging to the lesion region.

[0063] After obtaining the sample encoding feature map, the sample encoding feature map is divided into three branches: a lesion region segmentation branch, a three-dimensional brain midline segmentation branch, and an ideal brain midline surface positioning branch. Each branch reconstructs the features based on the sample encoding feature map to output the prediction results corresponding to each branch.

[0064] For the lesion region segmentation branch, in one possible implementation, lesion region prediction is performed based on the sample encoding feature map to predict the probability of each pixel point in the sample three-dimensional brain image belonging to the lesion region, and a single-channel first sample probability map is output. Corresponding to the three-dimensional brain image, the first sample probability map is also a three-dimensional image, and each probability value in the first sample probability map represents the probability of the pixel point in the sample three-dimensional brain image belonging to the lesion region. The greater the probability value, the greater the probability that the pixel point is located in the lesion region of the sample three-dimensional brain image.

[0065] Optionally, the lesion area in the lesion area prediction can be a brain hematoma area, can be a brain tumor area, can be a brain abscess area, etc. The present embodiment does not limit the lesion area, and only needs to ensure that the lesion area is an intracranial space-occupying lesion area (since the intracranial space-occupying lesion area will affect the three-dimensional brain midline structure in the brain).

[0066] At step 303, brain midline segmentation prediction is performed based on the sample encoding feature map to obtain a second sample probability map. The second sample probability map is used to represent the probability of each pixel point in the sample three-dimensional brain image belonging to the left hemisphere region, the right hemisphere region, and the background region, respectively.

[0067] For the brain midline segmentation prediction branch, since the three-dimensional brain midline is an anatomical structure separating the left and right hemispheres, accurately predicting the left hemisphere region, the right hemisphere region, and the background region in the sample three-dimensional brain image can determine the three-dimensional brain midline based on the left hemisphere region and the right hemisphere region. Therefore, in one possible implementation, in the brain midline segmentation prediction branch, left hemisphere region prediction, right hemisphere region prediction, and background region prediction are performed based on the sample encoding feature map to determine the probability of each pixel point in the sample three-dimensional brain image belonging to the left hemisphere region, the right hemisphere region, and the background region, respectively, and obtain a second sample probability map with three channels. Corresponding to the sample three-dimensional brain image, the second sample probability map is also a three-dimensional image, and the three channels in the second sample probability map correspond to the left hemisphere region, the right hemisphere region, and the background region, respectively. The probability values of the three channels represent the probability of the pixel point in the sample three-dimensional brain image belonging to the left hemisphere region, the right hemisphere region, and the background region, respectively, and the larger the probability value, the greater the probability of the pixel point being in the left hemisphere region or the right hemisphere region or the background region.

[0068] At step 304, ideal brain midplane prediction is performed based on the sample encoding feature map to obtain a sample prediction heat map. The sample prediction heat map is used to indicate the position of the ideal brain midplane in the sample three-dimensional brain image.

[0069] For the ideal brain midplane prediction branch, ideal brain midplane prediction is performed based on the sample encoding feature map to predict the position of the ideal brain midplane in the sample three-dimensional brain image to obtain a sample prediction heat map. Corresponding to the sample three-dimensional brain image, the sample prediction heat map is also a three-dimensional image, which is used to represent the position of the ideal brain midplane in the sample three-dimensional brain image.

[0070] At step 305, the feature encoding network is jointly trained based on the first sample probability map, the first labeled segmentation image, the second sample probability map, the second labeled segmentation image, the sample prediction heat map, and the sample labeled heat map.

[0071] In order to realize the supervised training of the three prediction tasks, the standard segmentation results of the three task branches also need to be labeled respectively in the sample data preparation stage. The standard segmentation results include: a first labeled segmentation image, a second labeled segmentation image and a sample labeled heat map. The first labeled segmentation image labels the lesion area in the sample three-dimensional brain image, the second labeled segmentation image labels the left brain area, the right brain area and the background area in the sample three-dimensional brain image, and the sample labeled heat map labels the ideal brain midline in the sample three-dimensional brain image. Subsequently, the model can be supervised and trained based on the standard segmentation results and the predicted segmentation results.

[0072] Since the multiple prediction tasks share the sample encoding feature map output by the same feature encoding network, in a possible implementation, when the multi-task prediction results are obtained, the feature encoding network can be jointly trained based on the first sample probability map, the first labeled segmentation image, the second sample probability map and the second labeled segmentation image, the sample predicted heat map and the sample labeled heat map, so as to improve the feature extraction accuracy of the feature encoding network.

[0073] Optionally, in the above three prediction branches, the sample encoding feature map can be decoded by different feature decoders respectively, that is, the sample encoding feature map is decoded by the first feature decoder to obtain the first sample probability map, the sample encoding feature map is decoded by the second feature decoder to obtain the second sample probability map, and the sample encoding feature map is decoded by the third feature decoder to obtain the sample predicted heat map. Optionally, since the prediction tasks implemented by different feature encoders are different, the encoder structures of the first feature encoder, the second feature encoder and the third feature encoder are different, and the developer can set them to have different numbers of convolution layers based on the requirements.

[0074] As Figure 4As shown, it shows a schematic diagram of the model training process shown in an exemplary embodiment of the present application. A sample three-dimensional brain image 401 is input into a feature encoding network 402, and a sample encoding feature map 403 output by the feature encoding network 402 is obtained. In the lesion area prediction branch, the lesion area is predicted based on the sample encoding feature map 403, and a first sample probability map 404 representing the lesion area is obtained; in the brain midline segmentation prediction branch, the brain midline segmentation prediction is performed based on the sample encoding feature map 403, and a second sample probability map 406 representing the left hemisphere region, the right hemisphere region, and the background region is obtained; in the ideal brain midplane prediction branch, the ideal brain midplane is predicted based on the sample encoding feature map 403, and a sample prediction heat map 408 representing the location of the ideal brain midplane is obtained; and then based on the first sample probability map 404 and the first annotated segmentation image 405, the second sample probability map 406 and the second annotated segmentation image 407, the sample prediction heat map 408 and the sample annotated heat map 409, the feature encoding network 402 is trained.

[0075] In this embodiment, a first sample probability map is obtained by predicting the lesion area on the sample coding feature map, a second sample probability map is obtained by predicting the brain midline segmentation on the sample coding feature map, and a sample prediction heat map is obtained by predicting the ideal brain midplane on the sample coding feature map; and a feature coding network is trained based on the differences between the first sample probability map, the second sample probability map, the sample prediction heat map and the standard segmentation results, thereby improving the feature extraction accuracy of the feature coding network in the three tasks, and thereby assisting the prediction of the brain midline segmentation task.

[0076] When the feature encoding network is trained through three prediction tasks, the segmentation losses corresponding to the three prediction tasks can be calculated separately, and then the total segmentation loss of the three segmentation tasks can be determined to jointly train the feature encoding network and realize the multi-task supervision process of the feature encoding network.

[0077] exist Figure 3 On the basis of Figure 5 As shown, step 305 may include steps 501 to 504.

[0078] Step 501: Determine a first segmentation loss based on a first sample probability map and a first labeled segmentation image.

[0079] In order to achieve supervision of the lesion area segmentation task, in one possible implementation, by comparing the difference between the lesion area prediction result - the first sample probability map, and the lesion area annotation result - the first annotated segmentation image, a first segmentation loss is determined to supervise the training of the feature encoding network.

[0080] In an exemplary example, the first segmentation loss may be calculated as shown in formula (1).

[0081]

[0082] wherein, l ih represents the first segmentation loss (lesion area prediction loss), y represents the probability value of each pixel point in the first labeled segmentation image, represents the probability value corresponding to each pixel point in the first sample probability map.

[0083] Step 502, based on the second sample probability map and the second labeled segmentation image, determine the second segmentation loss.

[0084] Since the brain midline segmentation task is converted into left brain area prediction, right brain area prediction and background area prediction task, in order to realize the supervision of brain midline segmentation task, in one possible implementation, it is necessary to compare the difference between each brain area prediction result-second sample probability map and the labeled brain area prediction result-second labeled segmentation image, and determine the second segmentation loss to supervise the training of feature encoding network.

[0085] Optionally, the second segmentation loss can use the cross entropy loss function between the second sample probability map and the second labeled segmentation image.

[0086] Since the purpose of the brain midline segmentation task is to accurately segment the three-dimensional brain midline, and the three-dimensional brain midline is the intersection area of the left and right brain, in order to make the feature encoding network pay more attention to the brain midline area in the sample three-dimensional image and improve the segmentation accuracy near the brain midline, a distance-related weight coefficient is introduced in the calculation process of the second segmentation loss. Corresponding to an exemplary example, step 502 can include steps 502A-502C.

[0087] Step 502A, obtain the sample loss weight corresponding to each pixel point in the second sample probability map.

[0088] In order to further improve the segmentation accuracy near the brain midline, a higher sample loss weight needs to be assigned to the pixel points closer to the brain midline when calculating the second segmentation loss, so that the feature encoding network pays more attention to the segmentation loss of the brain midline area. Therefore, in one possible implementation, first, the sample loss weight corresponding to each pixel point in the second sample probability map is obtained, so that the sample loss weight corresponding to each pixel point is introduced in the subsequent loss calculation process.

[0089] Since the sample loss weight is related to the distance of the pixel point to the brain midline, the sample distance map corresponding to the sample three-dimensional image needs to be calculated first, and then based on the sample distance map, the sample loss weight corresponding to each pixel point is determined. Corresponding to an exemplary example, step 502A can further include steps 502A1-502A3.

[0090] Step 502A1 : Determine and annotate the 3D brain midline based on the second annotated segmented image.

[0091] To calculate the distance from each pixel to the brain midline, it is necessary to determine the position of the annotated 3D brain midline in the sample 3D brain image. In one possible implementation, the annotated 3D brain midline can be determined directly based on the second annotated segmented image.

[0092] Optionally, the three-dimensional brain midline may be manually pre-marked.

[0093] Step 502A2: Determine the sample distance between each pixel point in the second sample probability image and the marked three-dimensional brain centerline.

[0094] After the annotated 3D brain midline is determined, the sample distance from each pixel point in the second sample probability image to the annotated 3D brain midline can be calculated respectively. Since the second sample probability image corresponds to the sample 3D brain image, the sample distance from each pixel point in the sample 3D brain image to the annotated 3D brain midline can also be calculated respectively.

[0095] Step 502A3: Based on the sample distance, determine the sample loss weight corresponding to each pixel point. The sample loss weight is negatively correlated with the sample distance.

[0096] In order to achieve the goal that the larger the sample distance (the farther the pixel point is from the annotated 3D brain midline), the smaller the corresponding sample loss weight, in an exemplary example, the calculation method of the designed sample loss weight can be shown in formula (2).

[0097] W=exp((cD) / c) (2)

[0098] Where W represents the sample loss weight, c is a fixed value representing a point outside the brain, which can be set to c = 200 mm, and D represents the sample distance. Formula (2) shows that when the sample distance is larger (the farther the pixel is from the annotated 3D brain midline), (cD) is smaller, (cD) / c is also smaller, and the corresponding W is smaller. Conversely, if the sample distance is smaller (the closer the pixel is to the 3D brain midline), (cD) is larger, (cD) / c is also larger, and the corresponding W is larger; thus, the purpose of the sample loss weight being negatively correlated with the sample distance is achieved.

[0099] In one possible implementation, after obtaining the sample distance corresponding to each pixel point in the sample three-dimensional brain image or the first sample probability image, the sample distance can be substituted into formula (2) to calculate the sample loss weight corresponding to each pixel point.

[0100] like Figure 6As shown, it shows a schematic diagram of the distance map and the weight map shown in one exemplary embodiment of the present application. The distance map and the weight map are described by taking a two-dimensional image as an example. The original image 601 is a sample two-dimensional brain image. The distance of each pixel point from the brain midline is calculated to obtain the distance map 602. The sample loss weight is calculated based on the distance map 602 to obtain the weight map 603. The weight of the points in the area close to the brain midline in the weight map 603 is relatively large.

[0101] In step 502B, the brain region segmentation loss is determined based on the sample loss weight, the second sample probability map and the second labeled segmentation image.

[0102] Different from directly using the cross-entropy loss between the second sample probability map and the second labeled segmentation image (in which the weight of each pixel point is the same), in the present embodiment, different sample loss weights are assigned to different pixel points according to their distance from the brain midline when calculating the segmentation loss, so that the feature encoding network pays more attention to the loss of the pixel points in the midline area. In one possible implementation, the first device can determine the brain region segmentation loss according to the sample loss weight, the second sample probability map and the second labeled segmentation image.

[0103] In one exemplary example, the brain region segmentation loss can be calculated in the manner shown in formula (3).

[0104]

[0105] wherein, l hemi represents the brain region segmentation loss, W i represents the sample loss weight corresponding to the i-th pixel point, y i represents the probability value corresponding to the i-th pixel point in the second labeled segmentation image, represents the probability value corresponding to the i-th pixel point in the second sample probability image, and N represents the number of pixel points in the sample three-dimensional image.

[0106] In step 502C, the second segmentation loss is determined based on the brain region segmentation loss.

[0107] Optionally, when the brain region segmentation loss is obtained, the brain region segmentation loss can be directly determined as the second segmentation loss to participate in the subsequent model training process.

[0108] In order to further improve the accuracy of the brain midline prediction, in addition to indirectly comparing the prediction accuracy of the left brain region, the right brain region and the background region, the brain midline segmentation loss is also set, which is directly determined according to the predicted brain midline and the labeled brain midline. In another exemplary example, step 502 can further include steps 502D-502G.

[0109] Step 502D: Perform Sobel operator processing on the second sample probability map to determine the sample left brain hemisphere contour and the sample right brain hemisphere contour corresponding to the sample three-dimensional brain image.

[0110] Since the three-dimensional brain midline is the dividing area between the left and right hemispheres of the brain, theoretically, the junction of the left and right hemisphere contours is the three-dimensional brain midline. Correspondingly, in a possible implementation, the second sample probability map can be processed by the Sobel operator to determine the sample left and right hemisphere contours corresponding to the sample three-dimensional brain image, and then the sample three-dimensional brain midline is determined based on the sample left and right hemisphere contours.

[0111] Optionally, since the second sample probability map is a three-dimensional image, the Sobel operator should also use a 3DSobel operator.

[0112] Step 502E: Determine the sample's three-dimensional brain midline based on the sample's left brain outline and the sample's right brain outline.

[0113] Optionally, the intersection of the left and right hemispheres of the sample can be calculated to determine the sample's 3D brain midline. In an exemplary example, the calculation formula for determining the sample's 3D brain midline can be shown in formula (4).

[0114] ml=Conv3d(brain l )*Conv3d(brain r ) (4)

[0115] Among them, ml represents the sample three-dimensional brain centerline, Conv3d(brain l ) represents the left brain contour obtained by performing Sobel operator processing on the left brain region in the second sample probability map; Conv3d(brain r ) represents the right brain contour obtained by applying the Sobel operator to the right brain region in the second sample probability map. As can be seen from formula (4), the sample 3D brain midline can be obtained by multiplying the left and right brain contours.

[0116] Step 502F: Determine the brain midline segmentation loss based on the sample 3D brain midline and the standard 3D brain midline.

[0117] In one possible implementation, the mean square error between the sample 3D brain midline and the standard 3D brain midline is used as the brain midline segmentation loss to supervise the 3D brain midline segmentation results. In an illustrative example, the calculation formula for the brain midline segmentation loss can be shown as formula (5).

[0118]

[0119] Among them, lml represents a brain midline segmentation loss, y ml represents a standard three-dimensional brain midline, represents a sample three-dimensional brain midline, and N represents the number of pixel points.

[0120] Step 502G, determining a second segmentation loss based on the brain region segmentation loss and the brain midline segmentation loss.

[0121] In order to supervise the segmentation result from two dimensions of brain region segmentation and brain midline segmentation, in a possible implementation, the brain region segmentation loss and the brain midline segmentation loss can be respectively determined based on the second sample probability map and the second label probability map, and the sum of the brain region segmentation loss and the brain midline segmentation loss is determined as the second segmentation loss.

[0122] In order to further smooth the predicted brain midline and improve the visual effect of three-dimensional brain midline segmentation, a surface smoothing constraint can also be applied to the segmentation result. Corresponding to another exemplary example, step 502 can further include steps 502H-502J.

[0123] Step 502H, performing extreme value extraction processing on the sample three-dimensional brain midline to obtain sample midline three-dimensional coordinates corresponding to the sample three-dimensional brain midline.

[0124] In a possible implementation, when the sample three-dimensional brain midline (surface) is obtained, a differentiable Soft Argmax function is used to process the sample three-dimensional brain midline, and the segmentation probability map of the sample three-dimensional brain midline is converted into sample midline three-dimensional coordinates. The conversion formula is shown in formula (6).

[0125]

[0126] Wherein, softargmax(x) represents the sample midline three-dimensional coordinates obtained by processing, β is a constant, i and j represent the coordinates of the current element in the one-dimensional vector, because Soft-Argmax is similar to non-maximum suppression, the difference between the maximum value and the minimum value can be enlarged. x represents the weight, and x i represents the weight of the i-th element.

[0127] Step 502I, calculating a smoothing loss based on the sample midline three-dimensional coordinates to determine a brain midline smoothing loss.

[0128] Optionally, after obtaining the sample midline three-dimensional coordinates, a surface smoothing constraint is applied to the three-dimensional brain midline to obtain the brain midline smoothing loss. The formula of the brain midline smoothing loss can be shown in formula (7).

[0129]

[0130] Wherein, lsmooth represents a midline smoothness loss in the brain, represents a partial differential of the three-dimensional coordinates of the midline in the sample, where u refers to a surface (Surface) composed of all points, and Surface can be represented by a HxW matrix, and each element is equivalent to the height of the corresponding point from the bottom surface.

[0131] Step 502J, determining a second segmentation loss based on the brain region segmentation loss, the midline segmentation loss, and the midline smoothness loss.

[0132] In order to supervise the segmentation result from three dimensions of brain region segmentation, midline segmentation, and surface smoothness, in one possible implementation, the brain region segmentation loss, the midline segmentation loss, and the midline smoothness loss can be determined based on the second sample probability map and the second label probability map, and the sum of the brain region segmentation loss, the midline segmentation loss, and the midline smoothness loss is determined as the second segmentation loss.

[0133] In one exemplary example, the overall loss function corresponding to the midline segmentation prediction branch can be as shown in formula (8).

[0134] l real =l hemi +l ml +l smooth (8)

[0135] Wherein, l real represents the total loss corresponding to the midline segmentation prediction task, l hemi represents the brain region segmentation loss, l ml represents the midline segmentation loss, and l smooth represents the midline smoothness loss.

[0136] Step 503, determining a third segmentation loss based on the sample prediction heat map and the sample label heat map.

[0137] In order to achieve supervision on the ideal mid-surface prediction task, in one possible implementation, the third segmentation loss is determined based on the difference between the ideal mid-surface prediction result-sample prediction heat map and the ideal mid-surface label result-sample label heat map to supervise the training of the feature encoding network.

[0138] Step 504, training the feature encoding network based on the first segmentation loss, the second segmentation loss, and the third segmentation loss.

[0139] Optionally, after determining the task prediction loss corresponding to each branch, the feature encoding network is trained by combining the loss functions of multiple tasks, that is, the feature encoding network is trained based on the sum of the first segmentation loss, the second segmentation loss and the third segmentation loss, so that the model performs better in the tasks of three-dimensional brain midline segmentation and ideal brain mid-surface detection, and achieves better supervised learning effect.

[0140] In an exemplary example, the total loss function of the model is shown in formula (9).

[0141] l total =l ih +l real +L ideal (9)

[0142] Wherein, l total represents the total loss of the model, l ih represents the first segmentation loss (lesion area prediction loss), l real represents the second segmentation loss (brain midline segmentation loss), L ideal represents the third segmentation loss (ideal brain mid-surface prediction loss).

[0143] As Figure 7As shown, it shows a schematic diagram of a model training process shown in another exemplary embodiment of the present application. The sample three-dimensional brain image 701 is input into the feature encoding network 702 to obtain the sample encoding feature map 703 output by the feature encoding network 702. In the lesion area prediction branch, the lesion area prediction is performed based on the sample encoding feature map 703 to obtain the first sample probability map 704 representing the lesion area; in the brain midline segmentation prediction branch, the brain midline segmentation prediction is performed based on the sample encoding feature map 703 to obtain the second sample probability map 706 representing the left half brain area, the right half brain area and the background area; the sample three-dimensional brain midline 711 is obtained by processing the second sample probability map 706 by the three-dimensional Sobel operator; in the ideal brain midplane prediction branch, the ideal brain midplane prediction is performed based on the sample encoding feature map 703 to obtain the sample prediction heat map 708 representing the position of the ideal brain midplane; in the loss determination stage, the first segmentation loss 710 (lesion area prediction loss) is determined based on the first sample probability map 704 and the first labeled segmentation image 705; the brain region segmentation loss is determined based on the second sample probability map 706 and the second labeled segmentation image 707, the brain midline segmentation loss is determined based on the sample three-dimensional brain midline 711 and the labeled three-dimensional brain midline 712, the SoftArgmax processing is performed on the sample three-dimensional brain midline 711 to determine the curved surface smoothing loss 714, and the second segmentation loss 715 is further determined based on the curved surface smoothing loss 714, the brain midline segmentation loss and the brain region segmentation loss; the third segmentation loss 713 (ideal brain midplane prediction loss) is determined based on the sample prediction heat map 708 and the sample labeled heat map 709. Then, the feature encoding network 702 is trained based on the sum of the first segmentation loss 710, the second segmentation loss 715 and the third segmentation loss 713.

[0144] In this embodiment, the loss calculation process of each task branch is described respectively, so that the feature encoding network is comprehensively trained by combining the loss functions of multiple tasks, better supervised learning effect is achieved, and the prediction effect of the model in the three-dimensional brain midline segmentation and ideal brain midplane detection tasks is improved.

[0145] In the ideal brain midplane detection task, not only the plane features are extracted in the image space, but also the key point features representing the plane parameters are extracted in the Hough space to jointly detect the ideal brain midplane.

[0146] On the basis of Figure 3 , as shown in Figure 8 , step 304 can include steps 801-803, and step 503 can be replaced by step 804.

[0147] Step 801, based on the sample encoding feature map, semantic brain midplane prediction is performed to obtain a sample semantic heat map, and the sample semantic heat map is used to represent the probability of each pixel point belonging to the ideal brain midplane.

[0148] In the ideal brain midplane prediction branch, in order to improve the detection accuracy of the ideal brain midplane, semantic plane segmentation and Hough key point detection are performed in the image space and the Hough space. In a possible implementation, in the ideal brain midplane prediction branch, on the one hand, semantic brain midplane prediction can be performed based on the sample coding feature map to predict the probability of each pixel point in the sample three-dimensional brain image belonging to the ideal brain midplane, so as to obtain a sample semantic heat map. The sample semantic heat map corresponding to the sample three-dimensional brain image is also a three-dimensional image, and the value of each pixel point in the sample semantic heat map represents the probability of the pixel point belonging to the ideal brain midplane. The greater the pixel value, the greater the probability that the pixel point is located on the ideal brain midplane.

[0149] Step 802, performing ideal brain midplane prediction based on the sample coding feature map to obtain an intermediate sample feature map.

[0150] Step 803, performing Hough space transformation on the intermediate sample feature map to obtain a sample Hough key point heat map.

[0151] In addition to detecting the ideal brain midplane in the image space, a differentiable DHT is additionally introduced for ideal brain midplane detection. Through Hough control conversion, the problem of detecting the ideal brain midplane in the image space can be converted into the identification problem of a target key point in the Hough space (the parameter set corresponding to the target key point is used to indicate the ideal brain midplane). In a possible implementation, by performing ideal brain midplane prediction on the sample coding feature map to obtain an intermediate sample feature map, and then performing Hough space transformation on the intermediate sample feature map, a sample Hough key point heat map can be obtained. The brightest point (peak point) in the sample Hough key point heat map is the target key point, and the parameter corresponding to the target key point is the polar coordinate parameter of the ideal brain midplane.

[0152] Step 804, training the feature coding network based on the first sample probability map, the first labeled segmentation image, the second sample probability map, the second labeled segmentation image, the sample semantic heat map, the labeled semantic heat map, the sample Hough key point heat map, and the labeled Hough key point heat map.

[0153] After introducing the ideal brain midplane detection task in the image space and the Hough space, the dual-space supervision loss in the image space and the Hough space also needs to be introduced in the model training process. In a possible implementation, the feature coding network is trained based on the first sample probability map, the first labeled segmentation image, the second sample probability map, the second labeled segmentation image, the sample semantic heat map, the labeled semantic heat map, the sample Hough key point heat map, and the labeled Hough key point heat map.

[0154] In an exemplary example, step 804 can include steps 804A-804E.

[0155] At step 804A, a first segmentation loss is determined based on the first sample probability map and the first labeled segmentation image.

[0156] At step 804B, a second segmentation loss is determined based on the second sample probability map and the second labeled segmentation image.

[0157] The embodiments of steps 804A and 804B can refer to the determination processes of the first segmentation loss and the second segmentation loss in the above embodiments, which will not be repeated here.

[0158] At step 804C, an image space segmentation loss is determined based on the sample semantic heat map and the labeled semantic heat map.

[0159] In the ideal brain surface prediction branch, the image space segmentation loss is determined by comparing the difference between the sample semantic heat map and the labeled semantic heat map, so as to introduce the supervision loss in the image space. The calculation formula of the image space segmentation loss can be shown in formula (10).

[0160]

[0161] Wherein, L spatial represents the image space segmentation loss, N represents the number of pixel points in the sample three-dimensional image, P out represents the sample semantic heat map, P gt represents the labeled semantic heat map.

[0162] Since the pixel value in the sample semantic heat map represents the probability of the pixel point being located on the ideal brain surface, for the pixel points with small probability values, if they participate in the loss calculation process, the calculation speed will be obviously reduced. In order to improve the calculation speed, the probability threshold is set for screening, and only the pixel points with high probability values are used for loss calculation. In a demonstrative example, step 804C can further include steps one and two.

[0163] Step one: based on the target probability threshold, the sample semantic heat map is screened to obtain a target pixel point set, and the probability value corresponding to the pixel points contained in the target pixel point set is greater than the target probability threshold.

[0164] Wherein, the target probability threshold is used to eliminate the pixel points with low probability values from the sample semantic heat map. The target probability threshold can be set by the developer, and the target probability threshold can be 0.1, for example.

[0165] In a possible implementation, based on the target probability threshold, the sample semantic heat map is screened to remove the pixel points with probability values lower than the target probability threshold in the sample semantic heat map, and a target pixel point set is obtained.

[0166] Step two, determine the loss between the target pixel point set and the labeled semantic heat map, and determine the image space segmentation loss.

[0167] Optionally, after obtaining the target pixel point set, when calculating the loss between the sample semantic heat map and the labeled semantic heat map, only the loss of each pixel point in the target pixel point set between the sample semantic heat map and the labeled semantic heat map can be calculated to determine the image space segmentation loss. By screening the target pixel point set, the calculation efficiency can be greatly improved.

[0168] Step 804D, based on the sample Hough key point heat map and the labeled Hough key point heat map, determine the Hough space segmentation loss.

[0169] In the ideal brain surface prediction branch, the Hough space segmentation loss is also determined by comparing the difference between the sample Hough key point heat map and the labeled Hough key point heat map to train the model to introduce the supervision loss of the Hough space. The calculation formula of the Hough space segmentation loss can be as shown in formula (11).

[0170]

[0171] Wherein, L Hough represents the Hough space segmentation loss, M represents the number of pixel points in the Hough space, H out represents the sample Hough key point heat map, H gt represents the labeled Hough key point heat map.

[0172] Optionally, the third segmentation loss is the sum of the Hough space segmentation loss and the image space segmentation loss. As shown in formula (12).

[0173] L ideal = L spatial + L Hough (12)

[0174] Wherein, L ideal represents the third segmentation loss (ideal brain surface prediction loss), L spatial represents the image space segmentation loss, and L Hough represents the Hough space segmentation loss.

[0175] Step 804E, based on the first segmentation loss, the second segmentation loss, the image space segmentation loss and the Hough space segmentation loss, train the feature encoding network.

[0176] Optionally, the feature encoding network is trained by combining the loss functions of multiple tasks, that is, based on the sum of the first segmentation loss, the second segmentation loss, the image space segmentation loss and the Hough space segmentation loss.

[0177] As Figure 9As shown in FIG. 9, which shows a process diagram of model training according to another example embodiment of the present application. A sample three-dimensional brain image 901 is input into a feature encoding network 902 to obtain a sample encoding feature map 903 output by the feature encoding network 902. In the lesion area prediction branch, the sample encoding feature map 903 is subjected to lesion area prediction by a first feature decoding network 904 to obtain a hematoma segmentation result 908 (taking the lesion area as the hematoma area as an example), and based on the hematoma segmentation result 908 and the labeled segmentation result, a hematoma segmentation loss 909 is determined; in the brain midline segmentation prediction branch, the sample encoding feature map 903 is subjected to brain midline segmentation prediction by a second feature decoding network 905 to obtain a hemibrain area segmentation result 910 (a second sample probability map); the hemibrain area segmentation result 910 is processed by a three-dimensional Sobel operator to obtain a brain midline segmentation result 911, and the brain midline segmentation result 911 is subjected to SoftArgmax processing to determine a curved surface smoothing loss 912; in the ideal brain midplane prediction branch, on the one hand, the sample encoding feature map 903 is subjected to semantic plane prediction by a third feature decoding network 906 to output a semantic plane heat map 913 representing the position of the ideal brain midplane, so as to determine a semantic plane loss 914; on the other hand, the sample encoding feature map 903 is subjected to ideal brain midplane prediction and depth Hough space conversion by a fourth feature decoding network 907 to output a Hough key point heat map 915, so as to determine a Hough plane loss 916.

[0178] In this embodiment, the ideal brain midplane is detected in the image space and the Hough space respectively, and a double-space supervision loss in the image space and the Hough space is introduced into the loss, and the semantic plane heat map and the Hough key point heat map are respectively constrained and supervised in the image space and the Hough space, so that the detection result of the ideal brain midplane is more accurate.

[0179] The above embodiment mainly describes the process of training the model by the multi-task loss function, and this embodiment mainly describes how to obtain the three-dimensional brain midline and the ideal brain midplane based on the model trained above in the model application stage, so as to further determine the brain midline offset.

[0180] Please refer to Figure 10 , which shows a flowchart of a three-dimensional brain midline segmentation method according to another example embodiment of the present application. The method is applied to Figure 1 The second device 120 shown in FIG. 10 is taken as an example, and the method comprises the following steps.

[0181] In step 1001, a target three-dimensional brain image is input into a feature encoding network for feature encoding to obtain a target encoding feature map.

[0182] In the model application stage, the model including the three task branches can be directly deployed to the second device, or the business personnel can deploy at least one branch model of the three task branches to the second device according to the needs, so as to complete different prediction tasks. For example, the midline prediction branch and the ideal midplane prediction branch are deployed to the second device to realize the synchronous prediction of the three-dimensional midline and the ideal midplane, and the midline offset is determined according to the three-dimensional midline and the ideal midplane; or the lesion area prediction branch is deployed to the second device to realize the lesion area prediction task of the brain.

[0183] The task purpose in the embodiment is to predict the midline offset, and the midline offset needs to be determined according to the three-dimensional midline and the ideal midplane, so the ideal midplane prediction branch and the midline prediction branch trained are needed to be deployed to the second device. In a possible implementation, the target three-dimensional brain image to be predicted is input into the feature encoder for feature encoding, and the target encoding feature map can be obtained.

[0184] Step 1002, the target encoding feature map is subjected to midline segmentation prediction to obtain a target segmentation probability map, and the target segmentation probability map is used to represent the probability that each pixel point in the target three-dimensional brain image belongs to the left hemisphere region, the right hemisphere region and the background region.

[0185] After obtaining the target encoding feature map, in the midline prediction branch, the target encoding feature map is subjected to midline segmentation prediction to predict the probability that each pixel point in the target three-dimensional brain image belongs to the left hemisphere region, the right hemisphere region and the background region, and obtain a target segmentation probability map.

[0186] Step 1003, the target encoding feature map is subjected to ideal midplane prediction to obtain a target heat map, and the target heat map is used to indicate the position of the ideal midplane in the target three-dimensional brain image.

[0187] In the ideal midplane prediction branch, the target encoding feature map is subjected to ideal midplane prediction to predict the position of the ideal midplane in the target three-dimensional brain image, and obtain a target heat map.

[0188] Step 1004, the target brain midline offset is determined based on the target segmentation probability map and the target heat map.

[0189] Optionally, when the midline prediction result-target segmentation probability map and the ideal midplane prediction result-target heat map are obtained, the target three-dimensional brain image can be determined based on the target segmentation probability map and the target heat map.

[0190] Since the target segmentation probability map can only represent the location of each region in the three-dimensional brain image, it is also necessary to extract the target three-dimensional brain midline from the target segmentation probability map. In a possible example, step 1004 can further include steps 1004A-1004D.

[0191] Step 1004A, Sobel operator processing is performed on the target segmentation probability map to determine the target left brain profile and the target right brain profile corresponding to the target three-dimensional brain image.

[0192] Step 1004B, based on the target left brain profile and the target right brain profile, the target three-dimensional brain midline is determined.

[0193] Similar to the determination of the sample three-dimensional brain midline on the model training side, when determining the target three-dimensional brain midline based on the target segmentation probability map, Sobel operator processing is also performed on the target segmentation probability map to determine the target left brain profile and the target right brain profile, and then the target left brain profile and the target right brain profile are multiplied to determine the target three-dimensional brain midline.

[0194] Step 1004C, based on the target heat map, the target ideal brain midplane is determined.

[0195] The target heat map can be an image space heat map or a Hough space key point heat map, and the present embodiment does not constitute a limitation. In a possible implementation, the target ideal brain midplane can be directly determined according to the target heat map, for example, a plane formed by points with larger probability values in the image space heat map is determined as the target ideal brain midplane, or the target ideal brain midplane is determined according to the parameter set of the peak point in the Hough space key point heat map.

[0196] Step 1004D, the maximum vertical distance between the target three-dimensional brain midline and the target ideal brain midplane is determined as the target brain midline offset.

[0197] In a possible implementation, after the target three-dimensional brain midline and the target ideal brain midplane are determined, the vertical distance between each pixel point in the target three-dimensional brain midline and the target ideal brain midplane can be calculated, and the maximum vertical distance in the vertical distance is determined as the target brain midline offset.

[0198] As Figure 11As shown, it shows a schematic diagram of a determination process of a brain midline shift amount according to an example embodiment of the present application. A target three-dimensional brain image 1101 is input into a feature encoding network 1102 to obtain a target encoding feature map 1103 output by the feature encoding network 1102; brain midline segmentation prediction is performed based on the target encoding feature map 1103 to obtain a target probability map 1104 representing a left brain region, a right brain region and a background region, and the target probability map 1104 is processed by a three-dimensional Sobel operator to obtain a target three-dimensional brain midline 1105; at the same time, ideal brain midplane prediction is performed based on the target encoding feature map 1103 to obtain a target prediction heat map 1106 representing a position of the ideal brain midplane, and the target prediction heat map 1106 is processed to obtain a target ideal brain midplane 1107; and then a maximum vertical distance between the target three-dimensional brain midline 1105 and the target ideal brain midplane 1107 is determined as a target brain midline shift amount 1108.

[0199] In this embodiment, by deploying the trained brain midline segmentation prediction task and the ideal brain midplane detection task, the purpose of simultaneously determining the three-dimensional brain midline and the ideal brain midplane can be achieved, and further, a determination manner is provided for determining the brain midline shift amount, so that the determination efficiency of the brain midline shift amount can be improved.

[0200] Figure 12 is a structural block diagram of a three-dimensional brain midline segmentation device provided by an example embodiment of the present application, and the device comprises:

[0201] A feature encoding module 1201 is configured to perform feature encoding on a sample three-dimensional brain image by a feature encoder to obtain a sample encoding feature map.

[0202] A multi-task segmentation prediction module 1202 is configured to perform lesion region prediction, brain midline segmentation prediction and ideal brain midplane prediction based on the sample encoding feature map to obtain a lesion region prediction result, a brain midline segmentation prediction result and an ideal brain midplane prediction result.

[0203] A training module 1203 is configured to jointly train the feature encoder based on the lesion region prediction result, the brain midline segmentation prediction result and the ideal brain midplane prediction result.

[0204] Optionally, the multi-task segmentation prediction module 1202 is further configured to:

[0205] perform the lesion region prediction based on the sample encoding feature map to obtain a first sample probability map, and the first sample probability map is used to represent a probability that each pixel point in the sample three-dimensional brain image belongs to a lesion region.

[0206] perform the midline segmentation prediction based on the sample encoded feature map to obtain a second sample probability map, the second sample probability map being used to represent probabilities of each pixel point in the sample three-dimensional brain image belonging to a left hemisphere region, a right hemisphere region, and a background region respectively;

[0207] perform the ideal mid-sagittal plane prediction based on the sample encoded feature map to obtain a sample predicted heat map, the sample predicted heat map being used to indicate a position of the ideal mid-sagittal plane in the sample three-dimensional brain image;

[0208] The training module 1203 is further configured to:

[0209] jointly train the feature encoding network based on the first sample probability map, a first labeled segmentation image, the second sample probability map, a second labeled segmentation image, the sample predicted heat map, and a sample labeled heat map, the first labeled segmentation image being labeled with the lesion region in the sample three-dimensional brain image, the second labeled segmentation image being labeled with the left hemisphere region, the right hemisphere region, and the background region in the sample three-dimensional brain image, and the sample labeled heat map being labeled with the ideal mid-sagittal plane in the sample three-dimensional brain image.

[0210] Optionally, the training module 1203 is further configured to:

[0211] determine a first segmentation loss based on the first sample probability map and the first labeled segmentation image;

[0212] determine a second segmentation loss based on the second sample probability map and the second labeled segmentation image;

[0213] determine a third segmentation loss based on the sample predicted heat map and the sample labeled heat map;

[0214] train the feature encoding network based on the first segmentation loss, the second segmentation loss, and the third segmentation loss.

[0215] Optionally, the training module 1203 is further configured to:

[0216] obtain a sample loss weight corresponding to each pixel point in the second sample probability map;

[0217] determine a brain region segmentation loss based on the sample loss weight, the second sample probability map, and the second labeled segmentation image;

[0218] determine the second segmentation loss based on the brain region segmentation loss.

[0219] Optionally, the training module 1203 is further configured to:

[0220] determine a labeled three-dimensional brain midline based on the second labeled segmentation image;

[0221] determine sample distances from each pixel point in the second sample probability image to the labeled three-dimensional brain midline;

[0222] determine, based on the sample distances, a sample loss weight corresponding to each pixel point, the sample loss weight being in a negative correlation with the sample distance.

[0223] Optionally, the training module 1203 is further configured to:

[0224] perform Sobel operator processing on the second sample probability image to determine a sample left half-brain contour and a sample right half-brain contour corresponding to the sample three-dimensional brain image;

[0225] determine a sample three-dimensional brain midline based on the sample left half-brain contour and the sample right half-brain contour;

[0226] determine a midline segmentation loss based on the sample three-dimensional brain midline and a standard three-dimensional brain midline;

[0227] The determining, based on the brain region segmentation loss, of the second segmentation loss comprises:

[0228] determining the second segmentation loss based on the brain region segmentation loss and the midline segmentation loss.

[0229] Optionally, the training module 1203 is further configured to:

[0230] perform extreme value extraction processing on the sample three-dimensional brain midline to obtain sample midline three-dimensional coordinates corresponding to the sample three-dimensional brain midline;

[0231] perform smooth loss calculation based on the sample midline three-dimensional coordinates to determine a midline smooth loss;

[0232] The determining, based on the brain region segmentation loss and the midline segmentation loss, of the second segmentation loss comprises:

[0233] determining the second segmentation loss based on the brain region segmentation loss, the midline segmentation loss, and the midline smooth loss.

[0234] Optionally, the multi-task segmentation prediction module 1202 is further configured to:

[0235] perform semantic mid-surface prediction based on the sample encoding feature map to obtain a sample semantic heat map, the sample semantic heat map being used to represent a probability of each pixel point belonging to the ideal mid-surface;

[0236] perform ideal face prediction in a brain based on the sample encoded feature map to obtain an intermediate sample feature map;

[0237] perform Hough space transformation on the intermediate sample feature map to obtain a sample Hough key point heat map;

[0238] The training module 1203 is further configured to:

[0239] train the feature encoding network based on the first sample probability map, the first labeled segmentation image, the second sample probability map, the second labeled segmentation image, the sample semantic heat map, the labeled semantic heat map, the sample Hough key point heat map, and the labeled Hough key point heat map.

[0240] Optionally, the training module 1203 is further configured to:

[0241] determine a first segmentation loss based on the first sample probability map and the first labeled segmentation image;

[0242] determine a second segmentation loss based on the second sample probability map and the second labeled segmentation image;

[0243] determine an image space segmentation loss based on the sample semantic heat map and the labeled semantic heat map;

[0244] determine a Hough space segmentation loss based on the sample Hough key point heat map and the labeled Hough key point heat map;

[0245] train the feature encoding network based on the first segmentation loss, the second segmentation loss, the image space segmentation loss, and the Hough space segmentation loss.

[0246] Optionally, the training module 1203 is further configured to:

[0247] perform screening on the sample semantic heat map based on a target probability threshold to obtain a target pixel point set, a probability value corresponding to a pixel point included in the target pixel point set being greater than the target probability threshold;

[0248] determine a loss of the target pixel point set between the sample semantic heat map and the labeled semantic heat map to determine the image space segmentation loss.

[0249] Optionally, the apparatus further includes:

[0250] The feature encoding module 1201 is configured to input a target three-dimensional brain image into the feature encoding network to perform feature encoding to obtain a target encoded feature map.

[0251] a brain midline segmentation prediction module, configured to perform brain midline segmentation prediction on the target encoding feature map to obtain a target segmentation probability map, wherein the target segmentation probability map is used to represent the probability that each pixel in the target three-dimensional brain image belongs to the left brain region, the right brain region, and the background region;

[0252] a brain mid-face prediction module, configured to perform an ideal brain mid-face prediction on the target encoding feature map to obtain a target heat map, wherein the target heat map is used to indicate a position of the ideal brain mid-face in the target three-dimensional brain image;

[0253] A determination module is used to determine the target brain midline offset based on the target segmentation probability map and the target heat map.

[0254] Optionally, the determining module is further configured to:

[0255] Performing Sobel operator processing on the target segmentation probability map to determine the target left brain contour and the target right brain contour corresponding to the target three-dimensional brain image;

[0256] determining a target three-dimensional brain midline based on the target left brain contour and the target right brain contour;

[0257] determining a target ideal brain mid-face based on the target heat map;

[0258] The maximum vertical distance between the target three-dimensional brain midline and the target ideal brain midplane is determined as the target brain midline offset.

[0259] In summary, the embodiment of the present application provides a three-dimensional brain midline segmentation method: by feature encoding the sample three-dimensional brain image, and performing multi-task prediction (lesion area prediction, brain midline segmentation prediction and ideal brain midplane prediction) based on the feature encoding results (sample encoding feature map), since the lesion area will cause lateral displacement of the brain midline structure and the ideal brain midplane is close to the position of the three-dimensional brain midline, therefore, by additionally adding the lesion area prediction task and the ideal brain midplane prediction task, the feature extraction accuracy of the feature encoding network for the relevant features in the brain midline segmentation prediction can be improved, thereby further improving the accuracy of the brain midline segmentation prediction task.

[0260] Figure 13 is a schematic diagram showing the structure of a computer device according to an exemplary embodiment. The computer device 1300 may be Figure 1 The first device 110 may also be Figure 1the second device 120 in FIG. 1. The computer device 1300 includes a central processing unit (CPU) 1301, a system memory 1304, including a random access memory (RAM) 1302 and a read-only memory (ROM) 1303, and a system bus 1305 that couples the system memory 1304 to the central processing unit 1301. The computer device 1300 also includes a basic input / output system (I / O) 1306 that helps transfer information between elements within the computer device, and a mass storage device 1307 for storing an operating system 1313, application programs 1314, and other program modules 1315.

[0261] The basic input / output system 1306 includes a display 1308 for displaying information and an input device 1309, such as a mouse, keyboard, etc., for inputting information into the computer device. Both the display 1308 and the input device 1309 are connected to the central processing unit 1301 through an input / output controller 1310 that is connected to the system bus 1305. The basic input / output system 1306 can also include the input / output controller 1310 for receiving and processing input from a number of other devices, including a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1310 provides output to a display screen, printer, or other type of output device.

[0262] The mass storage device 1307 is connected to the central processing unit 1301 through a mass storage controller (not shown) that is connected to the system bus 1305. The mass storage device 1307 and its associated computer device readable medium provide non-volatile storage for the computer device 1300. That is, the mass storage device 1307 can include a computer device readable medium (not shown) such as a hard disk or a compact disk read-only memory (CD-ROM) drive.

[0263] Without loss of generality, the computer device readable medium can include computer device storage media and communication media. Computer device storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer device readable instructions, data structures, program modules or other data. Computer device storage media includes RAM, ROM, Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer device storage media described above can be embodied in any computer device storage media technology. The system memory 1304 and the mass storage device 1307 described above can be collectively referred to as memory.

[0264] According to various embodiments of the present disclosure, the computer device 1300 can also operate in connection with a remote computer device over a network such as the Internet. That is, the computer device 1300 can connect to the network 1311 through the network interface unit 1312 connected to the system bus 1305, or can connect to other types of networks or remote computer device systems (not shown) using the network interface unit 1312.

[0265] The memory further includes one or more programs stored in the memory, and the central processing unit 1301 implements all or part of the steps of the three-dimensional brain midline segmentation method described above by executing the one or more programs.

[0266] The present application also provides a computer readable storage medium, the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by a processor to implement the three-dimensional brain midline segmentation method provided by the method embodiment.

[0267] The present application provides a computer program product or computer program, which includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device execute the three-dimensional brain midline segmentation method provided by the method embodiment.

[0268] The above sequence numbers of the embodiments of the present application are only for description, and do not represent advantages or disadvantages of the embodiments.

[0269] A person of ordinary skill in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or can be instructed by a program to complete the related hardware, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk.

[0270] The above is only an optional embodiment of the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A three-dimensional brain midline segmentation method, characterized in that: The method comprises: Perform feature encoding on the sample three-dimensional brain image through the feature encoding network to obtain a sample encoding feature map; Performing lesion region prediction based on the sample coding feature map to obtain a first sample probability map, wherein the first sample probability map is used to represent the probability that each pixel point in the sample three-dimensional brain image belongs to the lesion region; Performing brain midline segmentation prediction based on the sample encoding feature map to obtain a second sample probability map, wherein the second sample probability map is used to represent the probability that each pixel point in the sample three-dimensional brain image belongs to the left hemisphere region, the right hemisphere region, and the background region; Perform semantic brain mid-face prediction based on the sample coding feature map to obtain a sample semantic heat map, wherein the sample semantic heat map is used to characterize the probability that each pixel point belongs to the ideal brain mid-face; perform ideal brain mid-face prediction based on the sample coding feature map to obtain an intermediate sample feature map; perform Hough space transformation on the intermediate sample feature map to obtain a sample Hough key point heat map, wherein the peak point in the sample Hough key point heat map is the target key point, and the parameter corresponding to the target key point is the polar coordinate parameter of the ideal brain mid-face; The feature encoding network is trained based on the first sample probability map, the first annotated segmented image, the second sample probability map, the second annotated segmented image, the sample semantic heat map, the annotated semantic heat map, the sample Hough key point heat map, and the annotated Hough key point heat map, wherein the first annotated segmented image is annotated with the lesion area in the sample three-dimensional brain image, and the second annotated segmented image is annotated with the left brain region, the right brain region, and the background region in the sample three-dimensional brain image.

2. The method according to claim 1, characterized in that The step of training the feature encoding network based on the first sample probability map, the first annotated segmented image, the second sample probability map, the second annotated segmented image, the sample semantic heat map, the annotated semantic heat map, the sample Hough key point heat map, and the annotated Hough key point heat map comprises: determining a first segmentation loss based on the first sample probability map and the first labeled segmented image; determining a second segmentation loss based on the second sample probability map and the second labeled segmented image; Determining an image space segmentation loss based on the sample semantic heat map and the annotated semantic heat map; Determining a Hough space segmentation loss based on the sample Hough keypoint heatmap and the annotated Hough keypoint heatmap; The feature encoding network is trained based on the first segmentation loss, the second segmentation loss, the image space segmentation loss, and the Hough space segmentation loss.

3. The method according to claim 2, characterized in that The determining a second segmentation loss based on the second sample probability map and the second labeled segmentation image includes: Obtaining a sample loss weight corresponding to each pixel point in the second sample probability map; determining a brain region segmentation loss based on the sample loss weight, the second sample probability map, and the second labeled segmentation image; Based on the brain region segmentation loss, the second segmentation loss is determined.

4. The method according to claim 3, characterized in that The obtaining of the sample loss weight corresponding to each pixel point in the second sample probability map includes: determining a labeled three-dimensional brain midline based on the second labeled segmented image; determining a sample distance between each pixel point in the second sample probability image and the marked three-dimensional brain midline; Based on the sample distance, the sample loss weight corresponding to each pixel point is determined, and the sample loss weight is negatively correlated with the sample distance.

5. The method according to claim 3, characterized in that The determining a second segmentation loss based on the second sample probability map and the second labeled segmented image further includes: Performing Sobel operator processing on the second sample probability map to determine a sample left hemisphere outline and a sample right hemisphere outline corresponding to the sample three-dimensional brain image; Determining a three-dimensional brain midline of the sample based on the left brain outline and the right brain outline of the sample; determining a brain midline segmentation loss based on the sample three-dimensional brain midline and the standard three-dimensional brain midline; The determining the second segmentation loss based on the brain region segmentation loss includes: The second segmentation loss is determined based on the brain region segmentation loss and the brain midline segmentation loss.

6. The method according to claim 5, characterized in that The determining a second segmentation loss based on the second sample probability map and the second labeled segmented image further includes: Performing extreme value extraction processing on the sample three-dimensional brain midline to obtain the sample midline three-dimensional coordinates corresponding to the sample three-dimensional brain midline; Performing smoothing loss calculation based on the three-dimensional coordinates of the sample midline to determine the brain midline smoothing loss; The determining the second segmentation loss based on the brain region segmentation loss and the brain midline segmentation loss includes: The second segmentation loss is determined based on the brain region segmentation loss, the brain midline segmentation loss, and the brain midline smoothing loss.

7. The method according to claim 2, characterized in that The determining of the image space segmentation loss based on the sample semantic heat map and the annotated semantic heat map includes: The sample semantic heat map is screened based on a target probability threshold to obtain a target pixel point set, wherein the probability values ​​corresponding to the pixels included in the target pixel point set are greater than the target probability threshold; Determine the loss of the target pixel set between the sample semantic heat map and the annotated semantic heat map to determine the image space segmentation loss.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: Inputting the target three-dimensional brain image into the feature encoding network for feature encoding to obtain a target encoding feature map; Performing brain midline segmentation prediction on the target encoding feature map to obtain a target segmentation probability map, wherein the target segmentation probability map is used to represent the probability that each pixel point in the target three-dimensional brain image belongs to the left brain region, the right brain region, and the background region; Performing an ideal brain mid-surface prediction on the target encoding feature map to obtain a target heat map, wherein the target heat map is used to indicate a position of the ideal brain mid-surface in the target three-dimensional brain image; A target brain midline offset is determined based on the target segmentation probability map and the target heat map.

9. The method according to claim 8, characterized in that Determining the target brain midline offset based on the target segmentation probability map and the target heat map includes: Performing Sobel operator processing on the target segmentation probability map to determine the target left brain contour and the target right brain contour corresponding to the target three-dimensional brain image; determining a target three-dimensional brain midline based on the target left brain contour and the target right brain contour; determining a target ideal brain mid-face based on the target heat map; The maximum vertical distance between the target three-dimensional brain midline and the target ideal brain midplane is determined as the target brain midline offset.

10. A three-dimensional brain midline segmentation device, characterized in that: The device comprises: A feature encoding module is used to perform feature encoding on the sample three-dimensional brain image through a feature encoding network to obtain a sample encoding feature map; A multi-task segmentation prediction module is used to predict the lesion area based on the sample coding feature map to obtain a first sample probability map, which is used to characterize the probability that each pixel point in the sample three-dimensional brain image belongs to the lesion area; perform brain midline segmentation prediction based on the sample coding feature map to obtain a second sample probability map, which is used to characterize the probability that each pixel point in the sample three-dimensional brain image belongs to the left hemisphere area, the right hemisphere area and the background area respectively; perform semantic brain mid-surface prediction based on the sample coding feature map to obtain a sample semantic heat map, which is used to characterize the probability that each pixel point belongs to the ideal brain mid-surface; perform ideal brain mid-surface prediction based on the sample coding feature map to obtain an intermediate sample feature map; perform Hough space transformation on the intermediate sample feature map to obtain a sample Hough key point heat map, the peak point in the sample Hough key point heat map is the target key point, and the parameters corresponding to the target key point are the polar coordinate parameters of the ideal brain mid-surface; A training module is used to train the feature encoding network based on the first sample probability map, the first annotated segmented image, the second sample probability map, the second annotated segmented image, the sample semantic heat map, the annotated semantic heat map, the sample Hough key point heat map, and the annotated Hough key point heat map, wherein the first annotated segmented image is annotated with the lesion area in the sample three-dimensional brain image, and the second annotated segmented image is annotated with the left brain region, the right brain region, and the background region in the sample three-dimensional brain image.

11. A computer device, characterized in that: The computer device includes: a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the three-dimensional brain midline segmentation method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the three-dimensional brain midline segmentation method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product stores a computer program, and the computer program is loaded and executed by a processor to implement the three-dimensional brain midline segmentation method according to any one of claims 1 to 9.

Images