Medical image three-dimensional reconstruction and visual display method and device, equipment and medium

Abstract

The invention relates to a medical image three-dimensional reconstruction and visual display method and device, equipment and a medium. According to the method, automatic construction from an original medical image to a functional three-dimensional anatomical model is realized in a mode of combining deep learning and a knowledge graph. Firstly, an original image is preprocessed and segmented, function labeling is carried out in combination with an anatomical knowledge graph, then a global structure is optimized through a graph neural network, and finally, a three-dimensional anatomical model is generated by fusing function connection information among tissues. According to the method, the automation degree and accuracy of three-dimensional anatomical modeling are improved, organic combination of anatomical structures and functions is achieved, and powerful support is provided for precise medical treatment and surgical planning.

Description

Technical Field

[0001] The present invention belongs to the field of medical image processing technology, and in particular relates to a method, device, equipment and medium for three-dimensional reconstruction and visualization of medical images. Background Art

[0002] Three-dimensional reconstruction and visualization methods for medical images are crucial in modern medicine. They can transform two-dimensional images into intuitive three-dimensional models, providing key support for disease diagnosis and surgical planning. This technology integrates multi-source imaging data to generate accurate anatomical structure displays, which not only improves doctors' diagnostic efficiency but also provides a reliable reference for complex surgeries. However, existing methods often face limitations when dealing with complex anatomical structures. Traditional reconstruction techniques usually focus on the segmentation and presentation of a single organ or tissue, making it difficult to effectively integrate multiple interrelated anatomical structures, resulting in a lack of integrity and functional relevance in the model.

[0003] In this field, the core challenge lies in how to achieve effective aggregation of scattered anatomical structures. Scattered imaging data need to be integrated into a functionally meaningful overall structure through intelligent algorithms, but the current aggregation process faces the problem of coordinating local and global structures. Accurate reconstruction of local structures requires the algorithm to accurately identify and segment small tissues. However, when expanded to a larger range of systematic reconstruction, the cumulative error of local accuracy may cause distortion of the overall model. This problem further derives another technical difficulty, namely how to incorporate anatomical knowledge into the aggregation process to express the functional connection between tissues. Simple morphological reconstruction cannot meet the needs of functional relationship evaluation in surgical planning, and aggregation without knowledge guidance often leads to insufficient functional expression of the model in complex scenarios.

[0004] Therefore, how to achieve accurate aggregation of local structures through intelligent algorithms in three-dimensional reconstruction and integrate anatomical knowledge to build a functionally complete system model has become a key issue in the field of three-dimensional reconstruction and visualization of medical images. Summary of the Invention

[0005] Based on this, it is necessary to provide a method, device, equipment and medium for three-dimensional reconstruction and visualization of medical images to address the above technical problems.

[0006] In a first aspect, the present application provides a method for three-dimensional reconstruction and visualization of medical images, comprising:

[0007] S1. Extract raw image data from multi-source medical images, process the raw image data using denoising and standardization techniques, and generate image datasets.

[0008] S2, using a deep convolutional neural network to perform local structure segmentation on the image dataset and obtain a segmentation result set;

[0009] S3. Based on the segmentation result set and in combination with the pre-established anatomical knowledge graph, the functional attributes of each anatomical structure are determined and a functional annotation set is generated;

[0010] S4. Perform global structural aggregation on the function annotation set through a graph neural network to obtain a first aggregation model;

[0011] S5. Obtaining inter-tissue functional connectivity rules from the anatomical knowledge graph, embedding the inter-tissue functional connectivity rules into the first aggregation model, and generating a second aggregation model containing functional connectivity information;

[0012] S6. Use volume rendering technology to visualize the global structure and functional connectivity of the second aggregated model to obtain a three-dimensional anatomical model.

[0013] In a second aspect, the present application further provides a medical image three-dimensional reconstruction and visualization display device, comprising:

[0014] The data preprocessing module is used to extract raw image data from multi-source medical images, process the raw image data using denoising and standardization techniques, and generate image datasets;

[0015] The segmentation module is used to perform local structure segmentation on the image dataset using a deep convolutional neural network to obtain a segmentation result set;

[0016] The functional attribute determination module is used to determine the functional attributes of each anatomical structure based on the segmentation result set and combine it with the pre-established anatomical knowledge map to generate a functional annotation set;

[0017] A global structure aggregation module is used to perform global structure aggregation on the function annotation set through a graph neural network to obtain a first aggregation model;

[0018] A functional connectivity embedding module is used to obtain inter-tissue functional connectivity rules from the anatomical knowledge graph, embed the inter-tissue functional connectivity rules into the first aggregation model, and generate a second aggregation model containing functional connectivity information;

[0019] The visualization display module is used to visualize the global structure and functional connection of the second aggregation model using volume rendering technology to obtain a three-dimensional anatomical model.

[0020] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements a method for three-dimensional reconstruction and visualization of medical images as described in the first aspect.

[0021] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for three-dimensional reconstruction and visualization of medical images as described in the first aspect.

[0022] The aforementioned method, apparatus, device, and medium for 3D reconstruction and visualization of medical images achieves the automatic construction of functional 3D anatomical models from raw medical images through a combination of deep learning and knowledge graphs. The raw images are first preprocessed and segmented, functionally annotated using an anatomical knowledge graph, and then the global structure is optimized using a graph neural network. Finally, functional connectivity information between tissues is integrated to generate a 3D anatomical model. This method not only improves the automation and accuracy of 3D anatomical modeling but also organically integrates anatomical structure and function, providing strong support for precision medicine and surgical planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 A schematic diagram of the process of a medical image 3D reconstruction and visualization method provided by the present invention;

[0025] Figure 2 A schematic diagram of a process for generating a second aggregation model including functional connectivity information in an optional embodiment of the present invention;

[0026] Figure 3 This is a structural schematic diagram of a medical image three-dimensional reconstruction and visualization display device provided by the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0028] refer to Figure 1 , which presents a flow chart of a method for three-dimensional reconstruction and visualization of medical images provided by the present application, the method comprising the following steps:

[0029] S1. Extract raw image data from multi-source medical images, process the raw image data using denoising and normalization techniques, and generate image datasets.

[0030] Specifically, data in today's medical imaging field comes from a wide range of sources, including but not limited to two-dimensional image sequences generated by multiple devices such as computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET). These raw image data are often subject to interference from various factors, such as device noise and patient physiological movement, resulting in varying data quality.

[0031] Denoising is a key step in improving data quality. For example, MRI data primarily exhibits Rayleigh-distributed noise. Adaptive wavelet transforms can be used to decompose image data into frequency bands and then adaptively adjust thresholds based on the energy characteristics of each frequency band. This effectively suppresses noise while preserving anatomical edge details to the greatest extent possible. However, CT data is significantly affected by Poisson noise. Therefore, a non-local mean filtering algorithm can be used to leverage the image's inherent redundant information to smooth noise while maintaining the continuity of tissue structure.

[0032] Standardization processing aims to eliminate data differences caused by different devices and different scanning parameters. Taking grayscale value standardization as an example, the grayscale value of each image pixel is mapped to a unified standardized grayscale range (such as 0-255). In this process, the typical grayscale range of human tissue (such as bone, soft tissue, fat, etc.) is referenced, and linear or nonlinear transformations are used to make image data from different sources comparable. At the same time, geometric standardization is also indispensable. The image is resampled and the pixel spacing is unified to ensure the accurate size of the reconstructed 3D model in the spatial coordinate system, laying the foundation for subsequent precise measurement and analysis.

[0033] S2. Use deep convolutional neural network to perform local structure segmentation on the image dataset and obtain the segmentation result set.

[0034] Specifically, deep convolutional neural networks (DCNNs), a prominent example of artificial intelligence in image processing, demonstrate exceptional performance in medical image segmentation tasks. Their core principle lies in the ability of convolutional kernels to automatically learn local features within an image. For example, when segmenting ventricular structures in brain MRI images, the convolutional kernels in the network's initial convolutional layers can capture subtle differences in grayscale values ​​between pixels at the ventricle's edge and those in the surrounding brain tissue, as well as local texture features (such as the smooth contours of the ventricular walls).

[0035] As the network depth increases, the convolution kernel size gradually expands, and the receptive field increases accordingly, enabling the understanding of more complex contextual information. In the application scenario of lung nodule segmentation in lung CT images, deep convolution kernels can comprehensively consider the spatial relationship between nodules and surrounding structures such as blood vessels and bronchi, distinguish between benign and malignant nodules (such as the lobulation and burr characteristics of malignant nodules), and thus more accurately outline the nodule boundaries.

[0036] During network training, a large number of previously annotated medical imaging datasets (including the boundaries and categories of various anatomical structures) can be used. A backpropagation algorithm combined with a loss function (such as the cross-entropy loss function) is used to calculate the error between the network's predicted segmentation results and the actual annotations. By iteratively adjusting the network weight parameters, the network gradually learns optimal feature extraction and segmentation capabilities. During training, data augmentation techniques (such as image rotation, scaling, translation, and noise addition) can also be applied to expand the dataset size and improve the network's generalization capabilities, enabling it to adapt to the image data segmentation needs of different patients and imaging conditions.

[0037] S3. Based on the segmentation result set and combined with the pre-established anatomical knowledge graph, the functional attributes of each anatomical structure are determined and a functional annotation set is generated.

[0038] Specifically, the anatomical knowledge graph is a highly structured presentation of medical knowledge. It organizes a large number of anatomical concepts in the form of a graph, where nodes represent anatomical structures (such as the liver, gallbladder, bile duct, etc.) and edges represent the functional connections between them (such as bile is produced in the liver, flows into the gallbladder through the bile duct for storage, and when eating, the gallbladder contracts to discharge bile into the duodenum to assist digestion).

[0039] In the present invention, the pre-constructed anatomical knowledge graph is associated with the image segmentation results through a semantic similarity algorithm. For example, for the liver contour obtained by abdominal CT image segmentation, the nodes related to the liver are first retrieved in the knowledge graph to extract their functional attributes (such as the liver has functions such as detoxification, protein synthesis, and glycogen storage, and obtains blood supply through the portal vein and hepatic artery, returns blood through the hepatic vein, and bile is secreted through the bile duct system). By analyzing the spatial features such as the position of the liver in the image, the adjacent relationship with the surrounding blood vessels (such as the portal vein, hepatic artery, hepatic vein), and the biliary system (bile duct, gallbladder), combined with the connection rules defined in the knowledge graph, the functional attributes of the liver are determined, and these attributes are stored in the form of structured data in a functional annotation set. Similarly, for the cardiac image segmentation results, according to the definitions of structures such as the atria, ventricles, and valves in the knowledge graph, their functional roles in the blood circulation are annotated (such as the left ventricle is responsible for pumping oxygenated blood into the aorta and transporting it to various tissues and organs throughout the body).

[0040] S4. Perform global structural aggregation on the function annotation set through a graph neural network to obtain the first aggregation model.

[0041] Specifically, graph neural networks (GNNs) have unique advantages in processing data structures with complex relationships. In the context of 3D reconstruction of medical images, segmented anatomical structures and their functional annotations are treated as nodes in a graph. The connections between nodes are constructed based on the actual proximity, connectivity, and functional synergy of anatomical structures in the human body.

[0042] The GNN aggregation process starts from the local neighborhood, and each anatomical structure node first collects information from its directly adjacent nodes. For example, in the 3D reconstruction of the pelvic region, the bladder node receives information from adjacent nodes such as the bilateral ureters (the upper end is connected to the kidneys and the lower end is connected to the bladder), the urethra (connecting the bladder to the outside of the body), and the surrounding blood vessels (such as the bladder artery and vein), and nerves (branches of the pelvic nerve plexus). This information includes the spatial coordinates and functional properties of the adjacent structures (such as the urinary function of the ureters and the blood supply function of the blood vessels). Through the defined aggregation function (such as summation, averaging, etc. combined with the weight matrix), the collected peripheral information is integrated into the bladder node's own feature vector, initially achieving local structure fusion.

[0043] Subsequently, the aggregation process gradually expands its scope to consider anatomical structure nodes at greater distances. For example, in the scenario of joint reconstruction of the pelvis and abdomen, the bladder node further receives information from organ nodes such as the intestine (small intestine, adjacent parts of the colon) and the uterus (female), and updates its own feature vector based on the relationship between them in terms of physiological functions (such as intestinal peristalsis affecting the position of the bladder, and the compression relationship of the uterus on the bladder during pregnancy) and anatomical support structures (ligament connections, etc.). After multiple rounds of information transmission and aggregation, the feature vector of each anatomical structure node contains its systemic association information with multiple organs throughout the body, and finally constructs a first aggregation model that fully reflects the connection between local and global anatomical functions of the human body. This model not only presents the three-dimensional morphology of the anatomical structure, but also implies the role and interaction mode of each structure in the human physiological function network.

[0044] S5. Obtain inter-tissue functional connectivity rules from the anatomical knowledge graph, embed the inter-tissue functional connectivity rules into the first aggregation model, and generate a second aggregation model containing functional connectivity information.

[0045] Specifically, the functional connectivity rules between tissues extracted from the anatomical knowledge graph can cover a variety of types. For example, the neural conduction pathway rules: in the three-dimensional reconstruction of the nervous system, starting from the cerebral cortex neuron cluster node, according to the nerve fiber connection path defined in the knowledge graph (such as connecting the left and right cerebral hemispheres through the corpus callosum, and nerve fibers projecting along specific white matter fiber bundles to deep structures such as the thalamus and basal ganglia, and then extending downward to the brainstem and spinal cord to achieve systemic neural signal regulation), these functional connectivity rules such as nerve conduction direction and speed characteristics (differences in conduction speed between different nerve fiber types, such as myelinated nerve fibers conducting faster than unmyelinated nerve fibers) are embedded into the corresponding neural structure part of the first aggregation model.

[0046] For example, the functional connection rules of blood circulation: in the three-dimensional reconstruction scenario of the cardiovascular system, starting from the left ventricle node, according to the knowledge graph, blood flows into the aorta through the aortic valve, and along the arterial branches at all levels (such as the coronary arteries supply blood to the heart itself, the carotid arteries supply blood to the brain, the renal arteries supply blood to the kidneys, etc.), the oxygenated blood is transported to tissues and organs throughout the body, and then returns to the superior and inferior vena cava through veins at all levels, and finally returns to the right atrium. The circulation path rules. At the same time, combined with the blood flow velocity and blood pressure change characteristics in different vascular segments (such as high aortic blood pressure and fast flow rate, low capillary blood pressure and flow rate are conducive to slow material exchange), these functional connection details are integrated into the model to generate a second aggregation model containing functional connection information. This model can intuitively present the complex functional connections between tissues and organs through neural signal transmission, blood circulation material transport, etc., providing a visual basis for in-depth understanding of the physiological function mechanism of the human body.

[0047] S6. Use volume rendering technology to visualize the global structure and functional connectivity of the second aggregated model to obtain a three-dimensional anatomical model.

[0048] Specifically, volumetric rendering technology is widely used in the field of three-dimensional visualization of medical images. Its core principle is to decompose the three-dimensional anatomical model into a large number of voxels (voxels are the basic volume units in three-dimensional space, similar to pixels in two-dimensional images). Each voxel contains a variety of attribute information, such as grayscale value (reflecting tissue density, such as bones with high grayscale values ​​appearing white; soft tissues with medium grayscale values ​​appearing gray; air with low grayscale values ​​appearing black), color value (can be assigned specific colors based on tissue type, such as green for the liver and yellow for easy distinction), and opacity (bone tissue has an opacity close to 1, appearing completely opaque; blood in blood vessels is relatively translucent, with an opacity between 0 and 1; air tissue has an opacity close to 0, appearing transparent).

[0049] During the rendering process, a ray casting algorithm is used. Starting from the perspective of a virtual observer, rays are emitted through the voxel space of the 3D anatomical model. When rays intersect with voxels, the light intensity attenuation and color mixing are calculated based on the voxel's grayscale, color, and opacity properties. For example, when visualizing a chest anatomical model, light passes through structures such as the chest wall soft tissue (semi-transparent, with a certain grayscale and skin color component), lung tissue (containing air, transparent and with a low grayscale), and the heart (opaque, red). By simulating the interaction between light and these different tissue voxels, a realistic 3D anatomical model image is ultimately synthesized on a 2D display device. This image can clearly display the global morphology, internal details, and functional connectivity between tissues of the anatomical structure (such as the distribution and direction of blood vessels between organs and the shuttle connections of nerves between tissues), providing an intuitive and accurate visual reference for medical diagnosis and surgical planning, and assisting doctors in gaining a deep understanding of the pathophysiological changes in complex anatomical regions.

[0050] The above-mentioned method for 3D reconstruction and visualization of medical images automatically constructs functional 3D anatomical models from raw medical images through a combination of deep learning and knowledge graphs. The raw images are first preprocessed and segmented, and functional annotation is performed using an anatomical knowledge graph. The global structure is then optimized using a graph neural network, and finally, functional connectivity information between tissues is integrated to generate a 3D anatomical model. This method not only improves the automation and accuracy of 3D anatomical modeling, but also organically integrates anatomical structure and function, providing strong support for precision medicine and surgical planning.

[0051] In an optional embodiment, S2 includes the following steps:

[0052] S21. Extract a target area image from the second image data set, and use a deep convolutional neural network to perform feature extraction on the target area image to obtain a local feature set.

[0053] Specifically, the extraction of target area images from image datasets can be based on anatomical regions or medical research focuses. For example, when studying brain lesions, brain tissue area images are extracted from head CT or MRI scans. These target area images are then input into a deep convolutional neural network (DCNN). The convolution layer of the DCNN extracts features by sliding the convolution kernel on the image. The size and number of the convolution kernel are variable, and can capture features of different scales and directions. At each layer of the network, the convolution operation generates feature maps that reflect features at different levels in the image, such as edges, textures, and shapes. Through multiple layers of convolution and pooling operations, the network gradually extracts local feature sets of the target area images. These feature sets contain key information about the target area.

[0054] S22. Based on the local feature set, a segmentation algorithm is used to perform pixel-level classification on the target area image to obtain a preliminary segmentation result.

[0055] Specifically, pixel-level classification of the target region image using a set of local features is the core of the segmentation process. Segmentation algorithms can be fully connected conditional random fields (CRFs) or fully connected Markov random fields (MRFs). For example, a fully connected CRF considers the relationships between pixels and their local features, classifying each pixel as belonging to a specific anatomical structure. By optimizing the parameters of the conditional random field, the probability distribution of each pixel belonging to a certain category can be obtained, thus obtaining a preliminary segmentation result.

[0056] S23. Smoothing and optimizing the segmentation boundary of the preliminary segmentation result to obtain an optimized segmentation result.

[0057] Specifically, the purpose of smoothing and optimizing the segmentation boundary of the preliminary segmentation result is to improve the quality of the segmentation result. A variety of methods can be used, such as contour-based evolutionary algorithms or region-based smoothing algorithms. The contour-based evolutionary algorithm adjusts the segmentation boundary to make it more consistent with the actual anatomical structure boundary. For example, by using the principle of the active contour model (Snake model), the segmentation boundary gradually converges to the real boundary of the anatomical structure under the combined action of internal and external forces. The region-based smoothing algorithm smoothes the boundary while maintaining the internal consistency of the region. After smoothing optimization, an optimized segmentation result is obtained.

[0058] S24. Extract boundary information of each anatomical structure based on the optimized segmentation result to generate a segmentation result set.

[0059] Specifically, based on the optimized segmentation results, the boundary information of each anatomical structure is extracted and a segmentation result set is generated. This boundary information can be extracted using edge detection algorithms, such as the Canny edge detection algorithm, which detects the boundaries between different anatomical structures in the segmentation results and stores these boundaries as pixel coordinates. This boundary information forms the basis of the segmentation result set, which contains the boundary definition of each anatomical structure and provides the necessary anatomical structure information for functional attribute determination and global structure aggregation in subsequent steps.

[0060] In an optional embodiment, S3 includes the following steps:

[0061] S31. Acquire functional attributes corresponding to anatomical structures in the anatomical knowledge graph, and obtain an attribute mapping table based on the functional attributes.

[0062] Specifically, functional attributes corresponding to the segmented anatomical structures are retrieved from the anatomical knowledge graph and an attribute mapping table is constructed. The anatomical knowledge graph is a structured knowledge base that contains detailed information about anatomical structures and their functional attributes. For example, when processing cardiac structures, the functional attributes of structures such as the atria, ventricles, and valves are extracted from the knowledge graph. For example, the atria are responsible for collecting blood, the ventricles are responsible for pumping blood out of the heart, and the valves control the unidirectional flow of blood. These attributes are organized into an attribute mapping table, which contains identifiers of anatomical structures and their corresponding functional attributes.

[0063] S32. If there are missing functional attributes in the attribute mapping table, a semantic analysis algorithm is used to deduce and supplement the missing functional attributes from the anatomical knowledge graph to update the attribute mapping table.

[0064] Specifically, when constructing the attribute mapping table, the functional attributes of certain anatomical structures may be missing. In this case, a semantic analysis algorithm can be used to deduce and supplement the missing functional attributes. The semantic analysis algorithm infers the missing functional attributes based on the semantic relationships in the knowledge graph, such as the role of the anatomical structure in the human system and its interaction with other structures. For example, if the functional attributes of a specific muscle are missing, the algorithm can infer its possible function, such as assisting joint movement or maintaining posture, by analyzing its connection with other muscles with known functions and its position in the locomotor system. Through this process, the attribute mapping table is updated and improved.

[0065] S33. Attribute mapping is performed according to the anatomical structure and the attribute mapping table to obtain preliminary functional annotations.

[0066] Specifically, the segmented anatomical structures are associated with the updated attribute mapping table, and attribute mapping operations are performed to obtain preliminary functional annotations. Each anatomical structure is assigned corresponding functional attributes, forming a preliminary functional annotation dataset. For example, the anatomical structure of the liver within the abdominal cavity is mapped to its functional attributes such as detoxification and synthesis, preliminarily determining its role in human metabolism.

[0067] S34. Use information integration technology to fuse the preliminary functional annotations with the contextual information in the anatomical knowledge graph to obtain optimized functional annotations.

[0068] Specifically, to improve the accuracy and completeness of functional annotations, information integration techniques can be used to fuse them with contextual information from the anatomical knowledge graph. Contextual information includes the hierarchical relationships of anatomical structures within the human body and their interactions with other systems. For example, the functional annotation of the kidney must not only consider its blood filtration function but also its connections with the urinary and endocrine systems. This integration of information yields an optimized functional annotation that more comprehensively reflects the actual physiological function of the anatomical structure.

[0069] S35 , screening annotations that meet a preset threshold from the optimized function annotations to obtain a function annotation set.

[0070] Specifically, the optimized functional annotations are screened using preset screening criteria (such as confidence thresholds for functional attributes and the degree of match with known anatomical knowledge). For example, only functional annotations with high confidence (e.g., confidence greater than 90%) and strong consistency with anatomical knowledge are retained to ensure the accuracy and reliability of the functional annotation set. This functional annotation set provides precise functional information support for subsequent global structural aggregation and functional connectivity analysis.

[0071] In an optional embodiment, S4 includes the following steps:

[0072] S41. Obtain matching relationships between functional attributes and anatomical structures from the functional annotation set, and obtain an initial matching set based on the matching relationships.

[0073] Specifically, the functional annotation set contains each anatomical structure and its corresponding functional attributes. By analyzing these matching relationships, we can gain a preliminary understanding of the functional role of anatomical structures in the human body. For example, in the cardiovascular system, the atria and ventricles of the heart correspond to different blood collection and pumping functions, respectively. These matching relationships form the basis of the initial matching set.

[0074] S42. Use a graph neural network to extract structural features of the anatomical structures in the initial matching set to obtain a structural feature set.

[0075] Specifically, GNNs can process graph-structured data, where anatomical structures serve as nodes in the graph, and the connections between nodes are based on their actual proximity and functional connections in the human body. Through the GNN's convolutional operations, each node (anatomical structure) aggregates information from its neighboring nodes, extracting a set of structural features. For example, for the anatomical structure of the lung, its structural feature set might include features such as the shape of the lung lobes, bronchial branching patterns, and vascular distribution.

[0076] S43. Based on the structural feature set, a global analysis of the anatomical structure is performed through a graph neural network to obtain a first aggregation model.

[0077] Specifically, GNNs use multi-layer information transmission and aggregation to enable each node to integrate information from the global graph structure. For example, when analyzing abdominal anatomy, the liver node not only integrates its own structural features but also incorporates information from surrounding nodes such as the biliary system and vascular system, thereby constructing a first-order aggregation model that reflects the interrelationships and functional connections between anatomical structures throughout the body. This model not only presents the three-dimensional morphology of the anatomical structure but also implicitly captures its role in the human body's functional network.

[0078] S44. If the structural aggregation result in the first aggregation model is consistent with the attribute mapping in the anatomical knowledge graph, then the supplementary functional attributes are obtained from the anatomical knowledge graph to obtain an extended attribute set.

[0079] Specifically, the first aggregation model is verified to verify whether the structural aggregation results are consistent with the attribute mapping in the anatomical knowledge graph. If they are consistent, the model accurately reflects the known functional properties of the anatomical structure. At this point, additional functional attributes are obtained from the anatomical knowledge graph to expand the attribute set. For example, when analyzing the nervous system, if the model accurately aggregates the connection relationship between neuronal clusters and nerve fiber bundles, the knowledge graph can be used to supplement the specific functional attributes of these neural structures in areas such as sensory conduction and motor control, further enriching the model's functional information.

[0080] S45. Through attribute mapping, the extended attribute set and the structural feature set are integrated to obtain an optimized matching set.

[0081] Specifically, the attribute mapping process accurately matches supplemented functional attributes to corresponding anatomical structures, ensuring that each anatomical structure possesses not only structural characteristics but also complete and detailed functional attributes. For example, for the kidney anatomical structure, its structural characteristics are precisely matched to its functional attributes in the urinary system, such as blood filtration and hormone secretion, to form an optimized matching relationship.

[0082] S46: Update the first aggregation model according to the optimized matching set.

[0083] Specifically, the updated model integrates the structural characteristics and complete functional properties of anatomical structures, more accurately reflecting the global relationships and functional characteristics of human anatomy. For example, the heart model not only displays the spatial relationships between structures such as the atria, ventricles, and valves, but also details their functional roles in blood circulation, such as the blood collection function of the atria, the blood pumping function of the ventricles, and the unidirectional blood control function of the valves. This provides more accurate data support for subsequent functional connectivity analysis and visualization.

[0084] refer to Figure 2 In an optional embodiment, embedding inter-tissue functional connectivity rules into the first aggregation model to generate a second aggregation model containing functional connectivity information includes the following steps:

[0085] S51. Based on the functional connection rules between organizations, the graph database query language is used to obtain node and edge data to obtain a set of structured connection rules.

[0086] Specifically, based on the functional connectivity rules between tissues, relevant node and edge data is retrieved through a graph database query language (such as Cypher) to form a structured connectivity rule set. Nodes represent anatomical structures, while edges represent the functional connectivity relationships between them. For example, in the functional connectivity rules of the nervous system, neuronal cluster nodes are connected by synaptic edges, which carry functional properties such as nerve conduction direction and speed. These connectivity rules are extracted from the graph database of the anatomical knowledge graph using a query language to obtain a structured connectivity rule set, providing the data foundation for subsequent vectorization processing.

[0087] S52. Use the Word2Vec algorithm to vectorize the structured connection rule set to generate a low-dimensional vector representation; based on the knowledge embedding technology, obtain the rule embedding vector set according to the low-dimensional vector representation.

[0088] Specifically, the Word2Vec algorithm is a method for converting text data into low-dimensional vector representations, which can capture the semantic information in the text. Through training, the Word2Vec algorithm converts text data (such as functional connectivity rule descriptions) into low-dimensional vector representations, capturing the semantic similarities and associations between rules. For example, nerve conduction rules and blood circulation rules may have different vector representations in the semantic space, reflecting their differences in functional connectivity characteristics. Then, based on knowledge embedding technology, these low-dimensional vector representations are further converted into rule embedding vector sets, so that each functional connectivity rule is stored in vector form.

[0089] S53. If the dimension of the rule embedding vector set exceeds a preset value, a principal component analysis algorithm is used to reduce the dimension of the rule embedding vector set to update the rule embedding vector set.

[0090] Specifically, if the dimension of the rule embedding vector set exceeds a preset value, principal component analysis (PCA) is used to reduce the dimensionality of the rule embedding vector set to reduce computational complexity and extract key features. PCA maps high-dimensional vectors to a low-dimensional space by finding the principal component directions of the data, while preserving as much key information as possible from the original data.

[0091] S54. Based on the rule embedding vector set, a graph neural network algorithm is used to aggregate node features and connection rules to obtain a second aggregation model containing functional connection information.

[0092] Specifically, based on the updated rule embedding vector set, the graph neural network (GNN) algorithm is used to aggregate the node features and connection rules in the first aggregation model. The GNN algorithm can process graph structure data and integrate the node features of the anatomical structure (such as structural feature sets, functional attributes, etc.) with the vector representation of the functional connection rules. For example, in the aggregation process of the cardiovascular system, the features of the heart node are combined with the vector of the vascular connection rules, and the feature vector of the heart node is updated to include connection rule information such as blood flow direction and pressure changes. After the above process, a second aggregation model containing functional connection information is obtained. This model not only presents the morphology and functional attributes of the anatomical structure, but also integrates the complex functional connection relationship between tissues, providing comprehensive data support for subsequent visualization and medical analysis.

[0093] The above-mentioned method for 3D reconstruction and visualization of medical images automatically constructs functional 3D anatomical models from raw medical images through a combination of deep learning and knowledge graphs. The raw images are first preprocessed and segmented, and functional annotation is performed using an anatomical knowledge graph. The global structure is then optimized using a graph neural network, and finally, functional connectivity information between tissues is integrated to generate a 3D anatomical model. This method not only improves the automation and accuracy of 3D anatomical modeling, but also organically integrates anatomical structure and function, providing strong support for precision medicine and surgical planning.

[0094] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0095] Based on the same inventive concept, embodiments of the present application also provide an apparatus for implementing the aforementioned method for 3D reconstruction and visualization of medical images. The solution provided by this apparatus is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the apparatus for 3D reconstruction and visualization of medical images provided below can be found in the aforementioned limitations of the method for 3D reconstruction and visualization of medical images, and will not be further elaborated here.

[0096] In an exemplary embodiment, Figure 3As shown, a medical image three-dimensional reconstruction and visualization display device 30 is provided, comprising:

[0097] The data preprocessing module 31 is used to extract original image data from multi-source medical images, process the original image data using denoising and standardization techniques, and generate an image data set.

[0098] The segmentation module 32 is used to perform local structure segmentation on the image data set using a deep convolutional neural network to obtain a segmentation result set.

[0099] The functional attribute determination module 33 is used to determine the functional attributes of each anatomical structure based on the segmentation result set and in combination with the pre-established anatomical knowledge graph to generate a functional annotation set.

[0100] The global structure aggregation module 34 is used to perform global structure aggregation on the function annotation set through a graph neural network to obtain a first aggregation model.

[0101] The functional connectivity embedding module 35 is used to obtain inter-tissue functional connectivity rules from the anatomical knowledge graph, embed the inter-tissue functional connectivity rules into the first aggregation model, and generate a second aggregation model containing functional connectivity information.

[0102] The visualization display module 36 is used to visualize the global structure and functional connections of the second aggregated model using a volume rendering technique to obtain a three-dimensional anatomical model.

[0103] Optionally, the segmentation module includes:

[0104] The feature extraction unit is used to extract the target area image from the second image data set, and use a deep convolutional neural network to extract features from the target area image to obtain a local feature set.

[0105] The pixel-level classification unit is used to perform pixel-level classification on the target area image based on the local feature set and adopt the segmentation algorithm to obtain the preliminary segmentation result.

[0106] The boundary smoothing optimization unit is used to smoothly optimize the segmentation boundary of the preliminary segmentation result to obtain an optimized segmentation result.

[0107] The anatomical structure boundary extraction unit is used to extract the boundary information of each anatomical structure according to the optimized segmentation result and generate a segmentation result set.

[0108] Optionally, the functional attribute determination module includes:

[0109] The attribute mapping table construction unit is used to obtain the functional attributes corresponding to the anatomical structure in the anatomical knowledge graph and obtain the attribute mapping table according to the functional attributes.

[0110] The attribute supplement unit is used to deduce and supplement the missing functional attributes from the anatomical knowledge graph using a semantic analysis algorithm if there are missing functional attributes in the attribute mapping table, so as to update the attribute mapping table.

[0111] The preliminary function mapping unit is used to perform attribute mapping according to the anatomical structure and the attribute mapping table to obtain preliminary function annotations.

[0112] The functional annotation fusion unit is used to use information integration technology to fuse the preliminary functional annotations with the contextual information in the anatomical knowledge graph to obtain optimized functional annotations.

[0113] The function annotation screening unit is used to screen annotations that meet a preset threshold from the optimized function annotations to obtain a function annotation set.

[0114] Optionally, the global structure aggregation module includes:

[0115] The matching relationship acquisition unit is used to obtain the matching relationship between the functional attributes and the anatomical structure from the functional annotation set, and obtain an initial matching set based on the matching relationship.

[0116] The structural feature extraction unit is used to extract structural features of the anatomical structures in the initial matching set using a graph neural network to obtain a structural feature set.

[0117] The global model generation unit is used to perform a global analysis of the anatomical structure through a graph neural network according to the structural feature set to obtain a first aggregation model.

[0118] The attribute extension unit is used to obtain supplementary functional attributes from the anatomical knowledge graph to obtain an extended attribute set if the structure aggregation result in the first aggregation model is consistent with the attribute mapping in the anatomical knowledge graph.

[0119] The feature integration unit is used to integrate the extended attribute set with the structural feature set through attribute mapping to obtain an optimized matching set.

[0120] The model updating unit is configured to update the first aggregation model according to the optimized matching set.

[0121] Optionally, the functional connectivity embedding module includes:

[0122] The rule acquisition unit is used to obtain node and edge data based on the functional connection rules between organizations using the graph database query language to obtain a structured connection rule set.

[0123] The rule embedding unit is used to vectorize the structured connection rule set using the Word2Vec algorithm to generate a low-dimensional vector representation; based on the knowledge embedding technology, the rule embedding vector set is obtained according to the low-dimensional vector representation.

[0124] The vector dimensionality reduction unit is used to reduce the dimensionality of the rule embedding vector set by using a principal component analysis algorithm if the dimension of the rule embedding vector set exceeds a preset value, so as to update the rule embedding vector set.

[0125] The aggregation model generation unit is used to aggregate node features and connection rules according to the rule embedding vector set using a graph neural network algorithm to obtain a second aggregation model containing functional connection information.

[0126] An embodiment of the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0127] An embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0128] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0129] The above-described embodiments merely represent several implementation methods of the embodiments of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the concept of the embodiments of the present application, and these modifications and improvements fall within the scope of protection of the embodiments of the present application.

Claims

1. A method for three-dimensional reconstruction and visualization of medical images, characterized in that: The method comprises: S1. extracting raw image data from multi-source medical images, processing the raw image data using denoising and normalization techniques, and generating an image dataset; S2. Using a deep convolutional neural network to perform local structure segmentation on the image dataset to obtain a segmentation result set; S3. Determine the functional attributes of each anatomical structure based on the segmentation result set and combine it with a pre-established anatomical knowledge map to generate a functional annotation set; S4. Performing global structural aggregation on the function annotation set through a graph neural network to obtain a first aggregation model; S5. Obtaining inter-tissue functional connectivity rules from the anatomical knowledge graph, embedding the inter-tissue functional connectivity rules into the first aggregation model, and generating a second aggregation model containing functional connectivity information; S6. Use volume rendering technology to visualize the global structure and functional connections of the second aggregated model to obtain a three-dimensional anatomical model.

2. The method according to claim 1, characterized in that The S2 includes: S21, extracting a target area image from the second image dataset, and performing feature extraction on the target area image using a deep convolutional neural network to obtain a local feature set; S22, using a segmentation algorithm to perform pixel-level classification on the target area image based on the local feature set to obtain a preliminary segmentation result; S23, performing smooth optimization on the segmentation boundary of the preliminary segmentation result to obtain an optimized segmentation result; S24. Extract boundary information of each anatomical structure based on the optimized segmentation result to generate the segmentation result set.

3. The method according to claim 1, characterized in that The S3 includes: S31, obtaining the functional attributes corresponding to the anatomical structure in the anatomical knowledge graph, and obtaining an attribute mapping table according to the functional attributes; S32. If there are missing functional attributes in the attribute mapping table, deriving and supplementing the missing functional attributes from the anatomical knowledge graph using a semantic analysis algorithm to update the attribute mapping table; S33, performing attribute mapping according to the anatomical structure and the attribute mapping table to obtain preliminary function annotations; S34, using information integration technology to fuse the preliminary function annotations with the context information in the anatomical knowledge graph to obtain optimized function annotations; S35 , screening annotations that meet a preset threshold from the optimized function annotations to obtain the function annotation set.

4. The method according to claim 1, wherein The S4 includes: S41, obtaining a matching relationship between the functional attribute and the anatomical structure from the functional annotation set, and obtaining an initial matching set based on the matching relationship; S42, extracting structural features of the anatomical structures in the initial matching set using a graph neural network to obtain a structural feature set; S43. Performing a global analysis of the anatomical structure using a graph neural network based on the structural feature set to obtain the first aggregation model. S44. If the structure aggregation result in the first aggregation model is consistent with the attribute mapping in the anatomical knowledge graph, obtaining supplementary functional attributes from the anatomical knowledge graph to obtain an extended attribute set; S45. Integrate the extended attribute set and the structural feature set through attribute mapping to obtain an optimized matching set; S46: Update the first aggregation model according to the optimized matching set.

5. The method according to any one of claims 1 to 4, characterized in that The step of embedding the inter-tissue functional connectivity rule into the first aggregation model to generate a second aggregation model containing functional connectivity information includes: S51. Based on the inter-organizational functional connection rules, a graph database query language is used to obtain node and edge data to obtain a structured connection rule set; S52, using the Word2Vec algorithm to vectorize the structured connection rule set to generate a low-dimensional vector representation; based on the knowledge embedding technology, obtaining a rule embedding vector set according to the low-dimensional vector representation; S53: If the dimension of the rule embedding vector set exceeds a preset value, a principal component analysis algorithm is used to reduce the dimension of the rule embedding vector set to update the rule embedding vector set; S54. According to the rule embedding vector set, a graph neural network algorithm is used to aggregate node features and connection rules to obtain the second aggregation model containing the functional connection information.

6. A medical image three-dimensional reconstruction and visualization display device, characterized in that: The device comprises: A data preprocessing module is used to extract raw image data from multi-source medical images, process the raw image data using denoising and standardization techniques, and generate an image dataset; A segmentation module, configured to perform local structure segmentation on the image dataset using a deep convolutional neural network to obtain a segmentation result set; a functional attribute determination module, configured to determine the functional attributes of each anatomical structure based on the segmentation result set and in combination with a pre-established anatomical knowledge graph, and generate a functional annotation set; A global structure aggregation module, configured to perform global structure aggregation on the function annotation set through a graph neural network to obtain a first aggregation model; a functional connectivity embedding module, configured to obtain inter-tissue functional connectivity rules from the anatomical knowledge graph, embed the inter-tissue functional connectivity rules into the first aggregation model, and generate a second aggregation model containing functional connectivity information; A visualization display module is used to visualize the global structure and functional connections of the second aggregation model using volume rendering technology to obtain a three-dimensional anatomical model.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images