Multi-modal medical image automatic segmentation method and system based on three-dimensional reconstruction model

Abstract

The invention belongs to the technical field of image segmentation, and discloses a multi-modal medical image automatic segmentation method and system based on a three-dimensional reconstruction model.The multi-modal medical image automatic segmentation method comprises the steps that multi-modal image data are acquired, and local features of the multi-modal image data are extracted; identifying a patient tissue structure in the multi-modal image data based on the local features; segmenting the tissue structure of the patient in the multi-modal image data by using a segmentation technology, and reconstructing a three-dimensional virtual model based on the tissue structure of the patient; and fusing a three-dimensional virtual model body drawing result and a three-dimensional virtual model surface drawing result to realize three-dimensional display of a segmentation result. According to the method, the influence of human errors and subjective judgment is reduced by identifying and segmenting the tissue structure, so that boundaries of various tissues are identified more accurately, automatic image segmentation can be completed in a short time, and subsequent diagnosis and analysis can be performed more quickly.

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation, and in particular to a method and system for automatic segmentation of multimodal medical images based on a three-dimensional reconstruction model. Background Art

[0002] Tissue segmentation is a key task in medical image processing, which involves identifying and separating different biological tissues from medical imaging data. This process is crucial for clinical diagnosis, surgical planning, disease monitoring, and treatment evaluation. Tissue segmentation helps doctors quickly identify lesions in complex medical images, and accurate tissue segmentation can guide doctors to develop more precise treatment plans, especially when healthy tissues need to be avoided.

[0003] Tissue segmentation is a common image post-processing problem. Many quantitative analyses and later diagnoses require accurate tissue segmentation. In the prior art, the segmentation of target organs is often achieved by manually outlining the target area. This method is time-consuming and labor-intensive. In particular, manual segmentation of three-dimensional medical images is a very labor-intensive process, and it is greatly affected by human subjective factors. The manual segmentation results often vary greatly due to the different subjective judgments of experts. Even the segmentation results of the same image by the same expert at different times and under different conditions are different.

[0004] Therefore, how to provide a multimodal medical image automatic segmentation method and system based on a three-dimensional reconstruction model is a problem that needs to be solved urgently. Summary of the invention

[0005] The embodiments of the present invention provide a method and system for automatic segmentation of multimodal medical images based on a three-dimensional reconstruction model to solve the problem in the prior art that image segmentation is affected by human subjective factors.

[0006] In order to have a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended to be a general review, nor is it intended to identify key / important components or to delineate the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a preface to the detailed description that follows.

[0007] According to a first aspect of an embodiment of the present invention, a method for automatic segmentation of multimodal medical images based on a three-dimensional reconstruction model is provided.

[0008] In one embodiment, the multimodal medical image automatic segmentation method based on the three-dimensional reconstruction model includes:

[0009] Acquire multimodal image data, extract local features of the multimodal image data, and identify the patient's tissue structure in the multimodal image data based on the local features;

[0010] Segment the patient's tissue structure in the multimodal image data using segmentation technology, and reconstruct a three-dimensional virtual model based on the patient's tissue structure;

[0011] The three-dimensional virtual model volume rendering result and the three-dimensional virtual model surface rendering result are integrated to realize the three-dimensional display of the segmentation result.

[0012] In one embodiment, the acquiring of multimodal image data, extracting local features of the multimodal image data, and identifying the patient tissue structure in the multimodal image data based on the local features includes:

[0013] Construct a neural network model based on the attention gating mechanism, and enhance the feature recognition ability of the neural network model through the attention coefficient;

[0014] Verify and train the network weights of the neural network model, perform iterative optimization on the neural network model, and obtain an optimized neural network model;

[0015] The multimodal image data is used as input of the optimized neural network model; local features of the multimodal image data are extracted by optimizing the neural network model to identify the patient's tissue structure in the multimodal image data.

[0016] In one embodiment, the segmentation of the patient tissue structure in the multimodal image data using the segmentation technology and the reconstruction of the three-dimensional virtual model based on the patient tissue structure includes:

[0017] The region growing algorithm is used to determine the segmentation target region, and the neural network post-processing model is established with the initial segmentation region as the initial growth point;

[0018] Introducing the cross-modal attention mechanism into the neural network post-processing model, the optimized neural network post-processing model is obtained to achieve feature synergy between the modalities in multimodal image data;

[0019] The patient's tissue structure is segmented using an optimized neural network post-processing model to obtain a segmentation mask; a three-dimensional virtual model based on the patient's tissue structure is reconstructed based on the segmentation mask.

[0020] In one embodiment, determining the segmentation target region by using a region growing algorithm includes:

[0021] Selecting grayscale values ​​in the multimodal image data, and taking pixel points representing the target area as initial seed points, to generate initial seed point grayscale values;

[0022] The gray value of the initial seed point is selected as a downward expansion value and an upward expansion value to obtain a tolerance interval of the gray value growth;

[0023] Take the initial seed point as the starting point to grow towards all adjacent points. If the grayscale value of the adjacent points of the initial seed point is within the tolerance interval, the current adjacent point will be included in the segmentation target area.

[0024] The current adjacent points are used as new seed points, and the adjacent points of the new seed points are added to the segmentation target area until no new seed points are generated, and the final segmentation target area is obtained.

[0025] In one embodiment, the fusing of the 3D virtual model volume rendering result and the 3D virtual model surface rendering result to realize the 3D display of the segmentation result includes:

[0026] An intermediate model is obtained based on the training process of the three-dimensional virtual model, and a segmentation effect of the intermediate model is volume rendered;

[0027] The patient's three-dimensional data is fused layer by layer in the intermediate model, and the three-dimensional display of the segmentation results is achieved in combination with the volume rendering results of the intermediate model.

[0028] In one embodiment, the training process based on the three-dimensional virtual model obtains an intermediate model, and volume rendering is performed on the segmentation effect of the intermediate model, including:

[0029] Temporarily storing model parameters in the 3D virtual model training process, and generating an intermediate model based on the 3D virtual model training process;

[0030] The segmentation effect of the intermediate model is used as the drawn data body, and conditional judgment is performed on the sampling points of the projected light in the drawn data body;

[0031] The cumulative value of the sampling points of the projected light in the rendered data volume is calculated, and the segmentation effect of the intermediate model is volume rendered based on the cumulative value of the sampling points to obtain the volume rendering result of the intermediate model.

[0032] In one embodiment, conditional judgment on the sampling point of the projected light within the rendered data volume includes:

[0033] Collect the sampling points of the projected light in the drawn data volume, and determine whether the value of the voxel point corresponding to the sampling point in the segmentation mask is 0:

[0034] If the value of the voxel point corresponding to the sampling point in the segmentation mask is 0, it means that the sampling point does not belong to the target area, and the sampling point is skipped directly;

[0035] If the value of the voxel point corresponding to the sampling point in the segmentation mask is not 0, it means that the sampling point belongs to the target area, and the intermediate model needs to be volume rendered.

[0036] In one embodiment, fusing the patient's three-dimensional data layer by layer in the intermediate model and realizing three-dimensional display of the segmentation result in combination with the volume rendering result of the intermediate model includes:

[0037] Establish a multimodal image data equalization model, and use the multimodal image data equalization model to fuse and display the patient's three-dimensional data layer by layer;

[0038] The patient's 3D data fusion results and the volume rendering results of the intermediate model are integrated, and coarse segmentation and fine segmentation are performed in sequence to improve the 3D display effect of the segmentation results.

[0039] In one embodiment, the multimodal image data equalization model includes a generation network, a discrimination network, and a reconstruction network;

[0040] The generative network optimizes the discriminant loss function by generating adversarial learning strategies, and the expression of the discriminant loss function is:

[0041]

[0042] Where Ldis represents the discriminant loss function; D(I real ) represents the classification result of the discriminant network D on the real image; D(I tar ) represents the classification result of the discriminant network D on the generated image; Pr represents the real data distribution; P t represents the generated data distribution; E represents the discrimination probability; I real Represents a real image; I tar Indicates the generated image.

[0043] In one embodiment, the calculation formula of the attention coefficient is:

[0044] a=σ(ψ(W x X+W q Q+b);

[0045] Where a represents the attention coefficient; X represents the encoder feature map; Q represents the decoder feature map; W x , W q All represent learnable weight parameters; b is a bias term; ψ(·) is a linear mapping; σ(·) is a nonlinear activation function;

[0046] According to a second aspect of an embodiment of the present invention, a multimodal medical image automatic segmentation system based on a three-dimensional reconstruction model is provided.

[0047] In one embodiment, the multimodal medical image automatic segmentation system based on the three-dimensional reconstruction model includes:

[0048] A tissue structure recognition module, used to acquire multimodal image data, extract local features of the multimodal image data, and recognize the patient's tissue structure in the multimodal image data based on the local features;

[0049] A tissue structure segmentation module, used to segment the patient's tissue structure in the multimodal image data using segmentation technology, and reconstruct a three-dimensional virtual model based on the patient's tissue structure;

[0050] The segmentation result display module is used to fuse the three-dimensional virtual model volume rendering result and the three-dimensional virtual model surface rendering result to realize the three-dimensional display of the segmentation result.

[0051] According to a third aspect of an embodiment of the present invention, a computer device is provided.

[0052] In one embodiment, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0053] According to a fourth aspect of embodiments of the present invention, a computer-readable storage medium is provided.

[0054] In one embodiment, the computer-readable storage medium stores a computer program, and the computer program implements the steps of the above method when executed by a processor.

[0055] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:

[0056] The multimodal medical image automatic segmentation method provided by the present invention reduces the influence of human errors and subjective judgments by identifying and segmenting tissue structures, thereby more accurately identifying the boundaries of various tissues, especially in complex three-dimensional spaces. At the same time, the large amount of time and labor required for manual outlining are reduced through automated processing, especially when processing large amounts or high-resolution three-dimensional medical images, so that automatic image segmentation can be completed in a short time, thereby enabling faster subsequent diagnosis and analysis.

[0057] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0059] Figure 1 is a flow chart of a multimodal medical image automatic segmentation method based on a three-dimensional reconstruction model according to an exemplary embodiment;

[0060] Figure 2 is a principle block diagram of a multimodal medical image automatic segmentation system based on a three-dimensional reconstruction model according to an exemplary embodiment;

[0061] Figure 3 The figure is a schematic diagram showing the structure of a computer device according to an exemplary embodiment. DETAILED DESCRIPTION

[0062] The following description and accompanying drawings fully illustrate the specific embodiments of this article so that those skilled in the art can practice them. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. The scope of the embodiments of this article includes the entire scope of the claims, as well as all available equivalents of the claims. Herein, the terms "first", "second", etc. are only used to distinguish one element from another, without requiring or implying any actual relationship or order between these elements. In fact, the first element can also be called the second element, and vice versa. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the structure, device or equipment including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also include elements inherent to such structure, device or equipment. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the structure, device or equipment including the elements. Each embodiment is described in a progressive manner herein, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other.

[0063] The terms "longitudinal", "lateral", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. in this document indicate the orientation or position relationship based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing this document and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In the description of this document, unless otherwise specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a mechanical connection or an electrical connection, it can also be the internal communication of two elements, it can be a direct connection, or it can be an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0064] As used herein, the term "plurality" means two or more than two, unless otherwise specified.

[0065] In this document, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B means: A or B.

[0066] In this article, the term "and / or" is a description of the association relationship between objects, indicating that three relationships may exist. For example, A and / or B means: A or B, or, A and B.

[0067] It should be understood that, although the various steps in the flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0068] Each module in the device or system of the present application can be implemented in whole or in part by software, hardware, or a combination thereof. The above modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to the above modules.

[0069] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0070] Figure 1 An embodiment of the multimodal medical image automatic segmentation method based on a three-dimensional reconstruction model of the present invention is shown.

[0071] In this optional embodiment, the multimodal medical image automatic segmentation method based on the three-dimensional reconstruction model includes:

[0072] Step S101, acquiring multimodal image data, extracting local features of the multimodal image data, and identifying the patient's tissue structure in the multimodal image data based on the local features;

[0073] Step S103, segmenting the patient's tissue structure in the multimodal image data using a segmentation technique, and reconstructing a three-dimensional virtual model based on the patient's tissue structure;

[0074] Step S105 , integrating the three-dimensional virtual model volume rendering result and the three-dimensional virtual model surface rendering result to realize three-dimensional display of the segmentation result.

[0075] In this optional embodiment, the acquiring of multimodal image data, extracting local features of the multimodal image data, and identifying the patient tissue structure in the multimodal image data based on the local features includes:

[0076] Construct a neural network model based on the attention gating mechanism, and enhance the feature recognition ability of the neural network model through the attention coefficient; verify and train the network weights of the neural network model, perform iterative optimization on the neural network model to obtain an optimized neural network model; use multimodal image data as input to the optimized neural network model; extract local features of the multimodal image data through the optimized neural network model to identify the patient's tissue structure in the multimodal image data.

[0077] In this optional embodiment, multimodal image information is obtained, and a neural network architecture based on the Attention-UNet framework (a neural network architecture based on an attention mechanism) is used. Specifically, attention refers to the attention mechanism used in neural networks, which can help the network focus on the most important parts of the image, thereby improving processing effects and accuracy. U-Net is a popular convolutional neural network (CNN) architecture that is widely used in image segmentation tasks.

[0078] In this optional embodiment, Attention-UNet improves the accuracy and sensitivity of the model for image segmentation by integrating the attention mechanism. It is particularly suitable for medical image segmentation tasks, such as liver and tumor segmentation. Compared with the standard UNet, Attention-UNet can better suppress irrelevant image areas while highlighting specific local features. Different from the skip link method of the traditional UNet framework, the present invention introduces an attention gating mechanism to selectively enhance important features while ignoring irrelevant features, realizing the dynamic selection and weighting of spatial and channel features, wherein the calculation formula of the attention coefficient is:

[0079] a=σ(ψ(W x X+W q Q+b);

[0080] Where a represents the attention coefficient; X represents the encoder feature map; Q represents the decoder feature map; W x , W q All represent learnable weight parameters; b is a bias term; ψ(·) is a linear mapping; σ(·) is a nonlinear activation function. This attention coefficient can measure the correlation between the encoder feature map and the decoder feature map, thereby achieving selective enhancement of important features, effectively extracting important local features such as boundaries and details, and accurately identifying subtle anatomical structures (tumors, blood vessels, bile ducts, etc.).

[0081] In this optional embodiment, after determining the model backbone network, further through the five-fold cross-validation method or validation set validation (depending on the specific data set), adjust the model hyperparameters, such as image segmentation block size (i.e., patch-size), training sample batch size (i.e., batch-size), loss function, feature map depth, etc., to achieve the best recognition effect. After determining the hyperparameters, the network weights are trained based on the training set data and the cross entropy loss function is used for continuous iterative optimization. The model can automatically achieve parameter optimization and learning in this process.

[0082] In this optional embodiment, the segmentation objects of the present invention mainly exist in two aspects: on the one hand, for the preoperative three-dimensional reconstruction model of abdominal organs, the present invention needs to segment important tissue structures from the patient's preoperative images. The present invention adopts a semi-automatic interactive segmentation method based on region growing to segment the tissue structures in a three-dimensional medical image; on the second hand, for the medical image segmentation task of abdominal organs, the present invention needs to segment the contours of organs and surrounding important tissue structures from the intraoperative short-axis ultrasound image in real time to achieve the subsequent registration between the intraoperative image and the preoperative model. With respect to the first aspect, the present invention segments important tissue structures in a three-dimensional CT image by using a semi-automatic interactive segmentation technology based on region growing. With respect to the second aspect, the present invention uses the continuous maximum flow segmentation technology of prior probability distribution and uses GPU for parallel computing, so it has a very high computational efficiency, so that the contours of organs and surrounding tissue structures can be segmented in real time in the intraoperative ultrasound image.

[0083] In this optional embodiment, the segmentation of the patient tissue structure in the multimodal image data by using the segmentation technology and reconstructing the three-dimensional virtual model based on the patient tissue structure includes:

[0084] The region growing algorithm is used to determine the segmentation target area, and the neural network post-processing model is established with the initial segmentation area as the initial growing point; the cross-modal cross-attention mechanism is introduced into the neural network post-processing model to obtain an optimized neural network post-processing model to achieve feature synergy between modalities in multimodal image data; the patient's tissue structure is segmented using the optimized neural network post-processing model and a segmentation mask is obtained; and a three-dimensional virtual model based on the patient's tissue structure is reconstructed based on the segmentation mask.

[0085] In this optional embodiment, determining the segmentation target region by using a region growing algorithm includes:

[0086] A grayscale value is selected from the multimodal image data, and the pixel point representing the target area is used as the initial seed point to generate the grayscale value of the initial seed point; the downward expansion value and the upward expansion value are selected for the grayscale value of the initial seed point to obtain a tolerance interval for the grayscale value growth; the initial seed point is used as the starting point to grow toward all adjacent points, and if the grayscale value of the adjacent point of the initial seed point is within the tolerance interval, the current adjacent point is included in the segmentation target area; the current adjacent point is used as a new seed point, and the adjacent points of the new seed point are included in the segmentation target area, and the process stops when no new seed points are generated, and the final segmentation target area is obtained.

[0087] In this optional embodiment, the segmentation technology in the present invention is a semi-automatic interactive segmentation of region growing, and its main contents include: the user first manually selects a pixel point in the image whose grayscale value best represents the target area as a seed point, and sets the grayscale value of the seed point to I; then the user manually selects the downward expansion value T1 and the upward expansion value T2, thereby obtaining a tolerance interval of the grayscale value growth [1-T1, 1+T2]; finally, the seed point is used as the starting point to grow to all adjacent points around it. If the grayscale value of a certain adjacent point is within the tolerance interval, the point is included in the segmentation target area, and the point is used as a new seed point. This process is repeated until no new seed points are generated. The area marked by the user is used as the initial growth point, and a multimodal neural network post-processing algorithm is established. Specifically, a cross-modal cross-attention mechanism is introduced to achieve inter-modal feature synergy, and its expression is:

[0088] Xattn=Attention(M1,M2,Q,K,V);

[0089] In the formula, M1 and M2 represent the image features obtained from different modalities such as CT images and magnetic resonance images, respectively; Q, K, and V represent the query vector, key vector, and value vector, respectively; Attention(·) represents the cross attention function. Through this operation, information complementarity and synergistic enhancement can be achieved between modalities. After iterative training of a large amount of patient data, the accuracy of automatic image segmentation can be improved, and the key anatomical structures of the fused data can be segmented for subsequent intraoperative fusion registration.

[0090] In this optional embodiment, the Dice value of the model segmentation accuracy is required to be more than 98% for solid organs and more than 90% for ductal tissues. The Dice index reflects the degree of overlap between the model segmentation results and the actual results. This index of 100% indicates perfect segmentation, and the neural network post-processing algorithm can obtain a segmentation model and segmentation mask with a higher Dice value, thereby achieving accurate reconstruction of the patient's three-dimensional virtual model. Based on the accurate segmentation mask, the patient's three-dimensional virtual model is finally reconstructed in the computer, reflecting the patient's abdominal organ structure in a true stereoscopic panoramic manner, and accurately locating the organ tumor location, size, surrounding lymph nodes and vascular invasion.

[0091] In this optional embodiment, the fusing of the three-dimensional virtual model volume rendering result and the three-dimensional virtual model surface rendering result to realize the three-dimensional display of the segmentation result includes:

[0092] An intermediate model is obtained based on the training process of the three-dimensional virtual model, and the segmentation effect of the intermediate model is volume rendered; the patient's three-dimensional data is fused layer by layer in the intermediate model, and the three-dimensional display of the segmentation result is achieved in combination with the volume rendering result of the intermediate model.

[0093] In this optional embodiment, the step of obtaining an intermediate model based on the training process of the three-dimensional virtual model and performing volume rendering on the segmentation effect of the intermediate model includes:

[0094] The model parameters in the 3D virtual model training process are temporarily stored, and an intermediate model is generated based on the 3D virtual model training process; the segmentation effect of the intermediate model is used as the rendered data volume, and conditional judgment is performed on the sampling points of the projected light in the rendered data volume; the cumulative values ​​of the sampling points of the projected light in the rendered data volume are calculated, and the segmentation effect of the intermediate model is volume rendered based on the cumulative values ​​of the sampling points to obtain the volume rendering result of the intermediate model.

[0095] In this optional embodiment, conditional judgment on the sampling point of the projected light within the rendered data volume includes:

[0096] Collect the sampling points of the projected light in the rendered data volume, and determine whether the value of the voxel point corresponding to the sampling point in the segmentation mask is 0: if the value of the voxel point corresponding to the sampling point in the segmentation mask is 0, it means that the sampling point does not belong to the target area, and the sampling point is skipped directly; if the value of the voxel point corresponding to the sampling point in the segmentation mask is not 0, it means that the sampling point belongs to the target area, and the intermediate model needs to be volume rendered.

[0097] In this optional embodiment, fusing the patient's three-dimensional data layer by layer in the intermediate model and realizing three-dimensional display of the segmentation result in combination with the volume rendering result of the intermediate model includes:

[0098] A multimodal image data equalization model is established, and the patient's three-dimensional data is fused and displayed layer by layer using the multimodal image data equalization model; the patient's three-dimensional data fusion results and the volume rendering results of the intermediate model are integrated, and coarse segmentation and fine segmentation are performed in sequence to improve the three-dimensional display effect of the segmentation results.

[0099] In this optional embodiment, the above segmentation process is an iterative and interactive segmentation. The intermediate model trained so far can be saved by temporarily storing the model parameters during the training process, and the segmentation effect of the intermediate model can be volume rendered. The segmentation results trained so far can be intuitively seen, and the camera angle of view and the segmentation area can be adjusted quickly and accurately. In GPU-accelerated volume rendering, the final cumulative value of the sampling points of the projected light in the rendered data volume is calculated. In this accumulation process, the segmentation result is used as a mask to guide the volume rendering, that is, before calculating each sampling point of the projected light in the original CT data, first determine whether the value of the voxel point with the same position as the point in the segmentation result (mask) is 0: if it is 0, it means that the point does not belong to the target area and does not need volume rendering, so the point is directly skipped; if it is not 0, it means that the point belongs to the target area and needs volume rendering. The previously accumulated light intensity I and the newly generated light intensity are unified and simplified into color information (represented by C), and the iterative equation for the front-to-back form is:

[0100] C out =C out +(1-α out )C in ;

[0101] α out =α out +(1-α out )α in ;

[0102] In the formula, α represents transparency and C represents color information.

[0103] In this optional embodiment, a ray tracing algorithm is used in combination with the latest Nvidia Ampere architecture to fuse and display the three-dimensional data layer by layer in real time, and render the patient's three-dimensional data in a computer according to the color transfer function and the transparency transfer function, providing an intuitive and three-dimensional information model to facilitate subsequent surgical navigation and surgical planning. By integrating the volume rendering based on pipeline propagation with the model surface rendering results based on rasterization rendering, the full anatomical information of the volume rendering and the precise modeling of the surface rendering can be taken into account to achieve the best display effect, which mainly includes:

[0104] A multimodal image data equalization model is established. The multimodal image equalization model includes a generation network, a discriminant network, and a reconstruction network. These two networks optimize the discriminant loss by generating adversarial learning strategies, so that the synthetic image output by the generation network is as similar as possible to the real image. The expression of the discriminant loss function is:

[0105]

[0106] Where Ldis represents the discriminant loss function; D(I real ) represents the classification result of the discriminant network D on the real image; D(I tar ) represents the classification result of the discriminant network D on the generated image; Pr represents the real data distribution; P t represents the generated data distribution; E represents the discrimination probability; I real Represents a real image; I tar Represents the generated image; the first term of the discriminant loss function describes the discriminant network's ability to identify real images, while the second term describes the discriminant network's ability to identify synthetic images; the reconstruction network is used to retain the original modality image information in the synthetic image.

[0107] In this optional embodiment, a two-stage U-Net deep neural network framework is designed to perform coarse and fine segmentation on the fused medical image to obtain accurate segmentation results. The basic structure of the two-stage U-Net network is mainly divided into two parts: a coarse segmentation UNet network and a fine segmentation Unet network. In the first stage, a coarse segmentation network is used to perform coarse segmentation on the downsampled low-resolution image; in the second stage, the original image is combined with the coarse segmentation result of the first stage, and the fine segmentation network is used to fine-tune it to further improve the accuracy of the segmentation. Using cross-validation technology and heuristic rules, and based on data features (image size, voxel size, anisotropy, etc.), the network structure, loss function, data enhancement method, and hyperparameters such as data batch size and learning rate in the training process are automatically optimized to achieve the optimal fusion effect, which can more accurately display areas of interest such as organs and tumors.

[0108] In this optional embodiment, the present invention adopts a U-Net model segmentation reconstruction network, including two parts, encoding and decoding, wherein the encoding part is a down-sampling network, the network starts from the input image, and after multiple "(convolution + activation) × N + pooling" operations, the size of the hidden layer is gradually reduced, and more abstract information in the image is gradually extracted; combined with the decoding network, from the abstract high-order information, through multiple "(convolution + activation) × N + anti-pooling" operations, the size of the hidden layer is gradually restored to the original image size.

[0109] Figure 2An embodiment of the multimodal medical image automatic segmentation system based on a three-dimensional reconstruction model of the present invention is shown.

[0110] In this optional embodiment, the multimodal medical image automatic segmentation system based on the three-dimensional reconstruction model includes:

[0111] The tissue structure recognition module 201 is used to obtain multimodal image data, extract local features of the multimodal image data, and recognize the patient tissue structure in the multimodal image data based on the local features;

[0112] The tissue structure segmentation module 203 is used to segment the patient tissue structure in the multimodal image data using a segmentation technique, and to reconstruct a three-dimensional virtual model based on the patient tissue structure;

[0113] The segmentation result display module 205 is used to fuse the 3D virtual model volume rendering result and the 3D virtual model surface rendering result to realize the 3D display of the segmentation result.

[0114] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store static information and dynamic information data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in the above method embodiment are implemented.

[0115] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0116] In addition, the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiment when executing the computer program.

[0117] In addition, the present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiment are implemented.

[0118] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0119] The present invention is not limited to the structures which have been described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A multimodal medical image automatic segmentation method based on a three-dimensional reconstruction model, characterized in that: The multimodal medical image automatic segmentation method comprises: Acquire multimodal image data, extract local features of the multimodal image data, and identify the patient's tissue structure in the multimodal image data based on the local features; Segment the patient's tissue structure in the multimodal image data using segmentation technology, and reconstruct a three-dimensional virtual model based on the patient's tissue structure; The three-dimensional virtual model volume rendering result and the three-dimensional virtual model surface rendering result are integrated to realize the three-dimensional display of the segmentation result.

2. The multimodal medical image automatic segmentation method based on a three-dimensional reconstruction model according to claim 1, characterized in that: The step of acquiring multimodal image data, extracting local features of the multimodal image data, and identifying the patient tissue structure in the multimodal image data based on the local features includes: Construct a neural network model based on the attention gating mechanism, and enhance the feature recognition ability of the neural network model through the attention coefficient; Verify and train the network weights of the neural network model, perform iterative optimization on the neural network model, and obtain an optimized neural network model; The multimodal image data is used as input of the optimized neural network model; local features of the multimodal image data are extracted by optimizing the neural network model to identify the patient's tissue structure in the multimodal image data.

3. The multimodal medical image automatic segmentation method based on three-dimensional reconstruction model according to claim 2, characterized in that: The method of segmenting the patient tissue structure in the multimodal image data by using the segmentation technology and reconstructing a three-dimensional virtual model based on the patient tissue structure includes: The region growing algorithm is used to determine the segmentation target region, and the neural network post-processing model is established with the initial segmentation region as the initial growth point; Introducing the cross-modal attention mechanism into the neural network post-processing model, the optimized neural network post-processing model is obtained to achieve feature synergy between the modalities in multimodal image data; The patient's tissue structure is segmented using an optimized neural network post-processing model to obtain a segmentation mask; a three-dimensional virtual model based on the patient's tissue structure is reconstructed based on the segmentation mask.

4. The multimodal medical image automatic segmentation method based on a three-dimensional reconstruction model according to claim 3, characterized in that: Determining the segmentation target region by using the region growing algorithm includes: Selecting grayscale values ​​in the multimodal image data, and taking pixel points representing the target area as initial seed points, to generate initial seed point grayscale values; The gray value of the initial seed point is selected as a downward expansion value and an upward expansion value to obtain a tolerance interval of the gray value growth; Take the initial seed point as the starting point to grow towards all adjacent points. If the grayscale value of the adjacent points of the initial seed point is within the tolerance interval, the current adjacent point will be included in the segmentation target area. The current adjacent points are used as new seed points, and the adjacent points of the new seed points are added to the segmentation target area until no new seed points are generated, and the final segmentation target area is obtained.

5. The multimodal medical image automatic segmentation method based on a three-dimensional reconstruction model according to claim 4, characterized in that: The fusing of the three-dimensional virtual model volume rendering result and the three-dimensional virtual model surface rendering result to realize the three-dimensional display of the segmentation result includes: An intermediate model is obtained based on the training process of the three-dimensional virtual model, and a segmentation effect of the intermediate model is volume rendered; The patient's three-dimensional data is fused layer by layer in the intermediate model, and the three-dimensional display of the segmentation results is achieved in combination with the volume rendering results of the intermediate model.

6. The multimodal medical image automatic segmentation method based on three-dimensional reconstruction model according to claim 5, characterized in that: The step of obtaining an intermediate model based on the training process of the three-dimensional virtual model and performing volume rendering on the segmentation effect of the intermediate model includes: Temporarily storing model parameters in the 3D virtual model training process, and generating an intermediate model based on the 3D virtual model training process; The segmentation effect of the intermediate model is used as the drawn data body, and conditional judgment is performed on the sampling points of the projected light in the drawn data body; The cumulative value of the sampling points of the projected light in the rendered data volume is calculated, and the segmentation effect of the intermediate model is volume rendered based on the cumulative value of the sampling points to obtain the volume rendering result of the intermediate model.

7. The multimodal medical image automatic segmentation method based on a three-dimensional reconstruction model according to claim 6, characterized in that: The conditional judgment of the sampling point of the projected light in the drawn data volume includes: Collect the sampling points of the projected light in the drawn data volume, and determine whether the value of the voxel point corresponding to the sampling point in the segmentation mask is 0: If the value of the voxel point corresponding to the sampling point in the segmentation mask is 0, it means that the sampling point does not belong to the target area, and the sampling point is skipped directly; If the value of the voxel point corresponding to the sampling point in the segmentation mask is not 0, it means that the sampling point belongs to the target area, and the intermediate model needs to be volume rendered.

8. The multimodal medical image automatic segmentation method based on a three-dimensional reconstruction model according to claim 7, characterized in that: The step of fusing the patient's three-dimensional data layer by layer in the intermediate model and realizing the three-dimensional display of the segmentation result in combination with the volume rendering result of the intermediate model includes: Establish a multimodal image data equalization model, and use the multimodal image data equalization model to fuse and display the patient's three-dimensional data layer by layer; The patient's 3D data fusion results and the volume rendering results of the intermediate model are integrated, and coarse segmentation and fine segmentation are performed in sequence to improve the 3D display effect of the segmentation results.

9. The multimodal medical image automatic segmentation method based on three-dimensional reconstruction model according to claim 8, characterized in that: The multimodal image data equalization model includes a generation network, a discrimination network and a reconstruction network; The generative network optimizes the discriminant loss function by generating adversarial learning strategies, and the expression of the discriminant loss function is: Where Ldis represents the discriminant loss function; D(I real ) represents the classification result of the discriminant network D on the real image; D(I tar ) represents the classification result of the discriminant network D on the generated image; Pr represents the real data distribution; P t represents the generated data distribution; E represents the discrimination probability; I real Represents a real image; I tar Indicates the generated image.

10. A multimodal medical image automatic segmentation system based on a three-dimensional reconstruction model, characterized in that: The multimodal medical image automatic segmentation system includes: A tissue structure recognition module, used to acquire multimodal image data, extract local features of the multimodal image data, and recognize the patient's tissue structure in the multimodal image data based on the local features; A tissue structure segmentation module, used to segment the patient's tissue structure in the multimodal image data using segmentation technology, and reconstruct a three-dimensional virtual model based on the patient's tissue structure; The segmentation result display module is used to fuse the three-dimensional virtual model volume rendering result and the three-dimensional virtual model surface rendering result to realize the three-dimensional display of the segmentation result.

Images