Rendering and displaying 3D representations of anatomical structures
Patent Information
- Application Number
- JP2023526953
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2020-11-05
- Filing Date
- 2021-11-01
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-11-01
AI Technical Summary
The challenge of accurately diagnosing congenital heart disease (CHD) in fetuses is due to insufficient expertise in obtaining and interpreting fetal heart images, leading to a diagnostic gap between theoretical and actual detection rates, which hinders timely intervention.
A computer-implemented method for generating and displaying a 3D representation of anatomical structures using 3D landmark models to identify and track anatomical landmarks, controlling rendering parameters such as cut planes, viewing directions, and lighting to enhance the visibility and assessment of anatomical features.
This approach provides an intuitive and accurate 3D display of fetal hearts, facilitating improved identification and assessment of cardiac abnormalities, thereby enhancing diagnostic accuracy and enabling timely interventions.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image rendering, and more particularly to the rendering and display of the representation of anatomical structures.
Background Art
[0002] Congenital heart disease (CHD) is one of the most common types of birth defects, affecting 1% of newborns. CHD may be asymptomatic during fetal life but causes significant morbidity and mortality after birth. The earlier CHD is diagnosed, the better the outcomes at birth and treatment options. In particular, effective in-utero therapies for specific CHD lesions, such as fetal aortic valvuloplasty for hypoplastic left heart syndrome, are becoming available and have been shown to significantly improve the natural history of the disease.
[0003] However, these potential benefits rely on accurate fetal ultrasound diagnosis of CHD. Even in developed countries where fetal ultrasound is accessible to everyone, the fetal diagnosis rate of CHD in the community is recognized to be 30 - 50%. However, theoretically, a thorough screening of the fetal heart should be able to detect 90% of abnormalities.
[0004] The main cause of this diagnostic gap is inadequate / inconsistent / uneven expertise regarding the acquisition of appropriate cardiac views and the interpretation of fetal heart images. This is mainly thought to be due to the diagnostic challenges presented by the small and rapidly beating fetal heart and the relatively low experience among caregivers with each specific type of congenital heart disease. The signs of heart disease are often subtle, can touch on various structures within the heart such as valves, septa, myocardium, and exist under multiple morphologies / associations (Fallot Tetralogy), requiring carefully targeted examinations.
Summary of the Invention
Problems to be Solved by the Invention
[0005] Therefore, an intuitive, accurate, dynamic, and interactive display of the fetal heart is necessary to facilitate the identification of cardiac abnormalities.
[0006] In particular, there is a clear need to improve the rendering and representation of anatomical structures such as the (fetal) heart in order to enhance clinicians' understanding and ability to identify and assess the patient's condition. [Means for solving the problem]
[0007] The present invention is defined by the claims.
[0008] According to one aspect of the present invention, a computer-aided method for generating and displaying a 3D representation of an individual's anatomical structure is provided.
[0009] This computer implementation method includes the steps of: acquiring image data of an anatomical structure; processing the image data to generate a 3D anatomical model of the anatomical structure; processing the image data and / or 3D anatomical model to acquire one or more sets of one or more 2D images, wherein each identified set of one or more 2D images represents a different predetermined view of the anatomical structure; identifying one or more predetermined anatomical landmarks within each set of one or more 2D images; processing one or more predetermined anatomical landmarks to generate a 3D landmark model; and processing the 3D anatomical model and 3D landmark model to render and display a 3D representation of the anatomical structure on an output display.
[0010] This disclosure recognizes that image data available for generating / rendering 3D representations of anatomical structures can also be used to generate 3D landmark models based on identified anatomical landmarks of the anatomical structures. It also recognizes that these 3D landmark models can be used to control or provide additional information regarding the 3D representation of anatomical structures to facilitate the assessment or analysis of an individual's condition.
[0011] Therefore, the present invention relates to a mechanism for performing 3D rendering and display by using a 3D landmark model to control how 3D representations of anatomical structures are rendered and / or displayed. This provides a novel mechanism for generating displays of 3D representations of anatomical structures. By taking anatomical landmarks into account, more clinically relevant renderings and displays of anatomical structures can be performed, providing clinicians with additional and / or more relevant data for evaluating subjects.
[0012] A method for rendering and displaying a 3D representation of anatomical structures is provided. The rendering and / or display is based on a 3D landmark model. This 3D landmark model identifies / represents the location of anatomical landmarks of the subject in 3D coordinate space.
[0013] Image data may include image data of an anatomical structure captured over at least one period of the periodic motion of the anatomical structure. Each set of one or more 2D images includes a sequence of 2D images representing at least one period of the periodic motion of the anatomical structure. The step of processing one or more predetermined anatomical landmarks includes the steps of tracking one or more predetermined landmarks in each sequence of 2D images and generating a 3D landmark model based on the one or more predetermined landmarks tracked in each sequence of 2D images.
[0014] Therefore, a 3D landmark model can identify changes in the position of anatomical landmarks in a subject over a period of time. This effectively tracks the position of anatomical landmarks in 3D space over time. This approach makes it easy to modify how the rendering / display of the 3D anatomical model is performed as the position of specific landmarks changes, and as the 3D representation changes over time, it can draw the clinician's attention to various types of information.
[0015] The steps of processing a 3D anatomical model and a 3D landmark model preferably include defining at least one rendering parameter based on the 3D landmark model. The at least one rendering parameter may include one or more of the following: a cross-section of the rendered 3D representation, a viewing direction of the rendered 3D representation, lighting parameters of the rendered 3D representation, one or more textures of the rendered 3D representation, and / or one or more colors of the rendered 3D representation.
[0016] This disclosure understands that by controlling specific rendering parameters based on a landmark model, the rendering of anatomical models can be made more relevant and provide clinicians with additional information. This facilitates the provision of specific anatomical views or the highlighting of specific anatomical features based on determined anatomical landmarks. This allows for the provision of additional information to the user that was previously unavailable, such as drawing attention to specific locations or regions of the rendering of anatomical structures.
[0017] Therefore, this embodiment provides a novel mechanism for rendering 3D representations of anatomical structures; that is, it provides a new and improved approach for image processing.
[0018] A cutting plane, sometimes called a "clipping plane" or "section plane," defines a plane in 3D space. When rendering, parts of the anatomical structure on one side of the cutting plane are rendered, while parts of the anatomical structure on the opposite side are not. This provides a view where parts of the anatomical structure are not displayed, making it easier to examine the interior of the anatomical structure.
[0019] The viewing direction defines the position of the virtual camera and controls the apparent rotation and / or position of the 3D representation of the anatomical structure relative to the display. Therefore, as the viewing direction changes, the apparent rotation and / or position of the 3D representation displayed on the output display changes.
[0020] Optionally, the step of processing one or more predetermined anatomical landmarks includes the step of mapping one or more predetermined landmarks in each set of one or more 2D images to a 3D coordinate system to generate a 3D landmark model.
[0021] In some examples, the step of displaying a 3D representation of an anatomical structure includes the steps of aligning a 3D landmark model and the 3D representation in the same 3D coordinate space, and controlling the rendering of the 3D representation of the anatomical structure based on the 3D landmark model of the anatomical structure.
[0022] The steps of rendering and displaying a 3D representation of an anatomical structure may include defining a viewing direction for the 3D representation of the anatomical structure based on a 3D landmark model and desired anatomical features and / or anatomical views, and rendering and displaying the 3D representation of the anatomical structure based on the defined viewing direction.
[0023] The step of rendering and displaying a 3D representation of an anatomical structure may include defining one or more cross-sections of the 3D representation of the anatomical structure based on a 3D landmark model and on desired anatomical features and / or anatomical views, and rendering and displaying the 3D representation of the anatomical structure based on the one or more defined cross-sections.
[0024] The method may further include receiving, at a user interface, a first user input indicating a desired anatomical feature or anatomical view. Thus, the user can define the content of the rendered 3D representation of the anatomical structure, i.e., the anatomical features to be included within the (rendered) 3D representation of the anatomical structure.
[0025] In some examples, the desired anatomical feature and / or anatomical view is one of a plurality of predetermined anatomical views of the anatomical structure. Optionally, the selection of the predetermined anatomical view may be made by the user. That is, the user can select one of the plurality of predetermined anatomical views. This approach facilitates the selection of an appropriate anatomical view for evaluating a patient's condition (e.g., for aligning an anatomical view provided by a 3D representation to an established anatomical view).
[0026] In other examples, the desired anatomical feature and / or anatomical view is a predetermined anatomical feature / view based on, for example, a standard procedure for viewing features / views for a particular diagnosis or analysis method.
[0027] The step of rendering and displaying a 3D representation of an anatomical structure may include receiving, at a user interface, a second user input indicating one or more desired viewing directions and / or cutting planes for the 3D representation of the anatomical structure; rendering and displaying the 3D representation of the anatomical structure based on the received second user input; using a 3D landmark model to predict anatomical features and / or anatomical views provided by the displayed 3D representation of the anatomical structure; and displaying information regarding the predicted anatomical features and / or anatomical views provided by the displayed 3D representation of the anatomical structure.
[0028] The anatomical structure may include a heart. The present invention is particularly advantageous when employed to provide a 3D representation of the heart. This is because the identification of anatomical heart landmarks is particularly well-developed. Thus, generating a 3D landmark model of the heart is a particularly suitable example. However, in other embodiments, the anatomical structure may include the brain, lungs, and the like.
[0029] In some examples, each identified set consisting of one or more 2D images represents a different one of a four-chamber view, a left ventricular outflow tract view, a right ventricular outflow tract view, a three-vessel view, a three-vessel trachea view, an abdominal view, an aortic arch view, and / or a ductus arteriosus arch view.
[0030] The step of processing image data and / or a 3D anatomical model may include processing the image data and / or the 3D anatomical model using a machine learning method to identify one or more sets consisting of one or more 2D images.
[0031] A computer program product including computer program code means is also proposed. When the computer program code means is executed on a computing device having a processing system, it causes the processing system to execute all the steps of any of the methods described herein.
[0032] A processing system for generating and displaying a 3D representation of an individual's anatomical structure has also been proposed. This processing system performs the following: acquiring image data of an anatomical structure; processing the image data to generate a 3D anatomical model of the anatomical structure; processing the image data and / or 3D anatomical model to acquire one or more sets of one or more 2D images, each identified set of one or more 2D images representing a different predetermined view of the anatomical structure; identifying one or more predetermined anatomical landmarks within each set of one or more 2D images; processing one or more predetermined anatomical landmarks to generate a 3D landmark model; and processing the 3D anatomical model and 3D landmark model to render and display a 3D representation of the anatomical structure on an output display.
[0033] Those skilled in the art will be able to easily adapt this processing system to perform any of the methods described herein. Similarly, those skilled in the art will be able to easily adapt the methods described herein to perform the operations of the processing system described herein.
[0034] These and other aspects of the present invention will become apparent from the embodiments described below and will be explained with reference to those embodiments. [Brief explanation of the drawing]
[0035] To gain a deeper understanding of the present invention and to more clearly illustrate how it is implemented, please refer to the attached drawings as just one example.
[0036] [Figure 1] Figure 1 shows the workflow performed by an embodiment of the present invention. [Figure 2] Figure 2 shows a set of 2D images of the fetal heart obtained from ultrasound image data over one cardiac cycle. [Figure 3]Figure 3 shows views of the left ventricle of the heart in a four-chamber view with different rendering parameters applied. [Figure 4] Figure 4 shows the mitral valve and tricuspid valve from valve opening to valve closing. [Figure 5] Figure 5 shows the processing system. [Modes for carrying out the invention]
[0037] The present invention will be described with reference to the figures.
[0038] The detailed descriptions and specific examples illustrate exemplary embodiments of the apparatus, system, and method, but should be understood to be for illustrative purposes only and not intended to limit the scope of the invention. These and other features, aspects, and advantages of the apparatus, system, and method of the present invention will be better understood from the following description, the appended claims, and the appended drawings. It should be understood that the figures are schematic diagrams and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the figures to indicate the same or similar parts.
[0039] This invention provides a method for generating and displaying a 3D representation of an individual's anatomical structure. Image data of the anatomical structure is acquired and processed to obtain a 2D image or sequence of images corresponding to a given view of the anatomical structure. Anatomical landmarks are identified within the 2D image or sequence of images and used to determine a 3D landmark model of the anatomical structure. The 3D representation of the anatomical structure is rendered and displayed using the 3D landmark model. This invention relies on the use of a 3D landmark model, also known as a "pose model," to control how the 3D representation of the anatomical structure is rendered and displayed. The 3D landmark model identifies the locations of anatomical landmarks in the anatomical structure.
[0040] For example, a 3D landmark model identifies the location of anatomical landmarks relative to a 3D anatomical model of an anatomical structure. The 3D anatomical model, when rendered appropriately, contains pre-rendered data that generates a 3D representation of the anatomical structure for display. Rendering and / or display of the 3D anatomical model is based on (i.e., dependent on) the 3D landmark model. In particular, one or more rendering or display parameters are defined using the 3D landmark model.
[0041] Therefore, the rendering of anatomical structures is based on the anatomical landmarks of the anatomical structure. This provides an approach to rendering and displaying representations of anatomical structures based on important anatomical aspects of the anatomical structure, enabling the rendering of a given anatomical view and / or a given part of the anatomical structure, thereby facilitating the assessment of the patient's condition.
[0042] Embodiments of the present invention can be used in any suitable 3D anatomical rendering scenario, such as fetal heart or brain evaluation. The present invention is particularly useful in fetal heart analysis, as this field is considered particularly challenging to perform accurate user analysis in.
[0043] Figure 1 schematically illustrates the workflow 100 performed by the embodiment in order to understand the underlying concepts of this disclosure.
[0044] In step 110, image data 191 of the anatomical structure is acquired. The image data is medical image data such as ultrasound image data, magnetic resonance image data, computed tomography image data, and X-ray image data. The image data is arbitrary data that, when processed, generates a 3D anatomical model of the anatomical structure (for example, generates a 3D image of the anatomical structure).
[0045] In process 120, image data 191 is processed to generate a 3D anatomical model 192 of the anatomical structure. The 3D anatomical model is a data representation of the anatomical structure, and when rendered and displayed, it generates a 3D representation (i.e., an image) of the anatomical structure. The 3D anatomical model provides information about the spatial location of various anatomical features or boundaries of the anatomical structure. Methods for generating a 3D anatomical model from image data are well known to those skilled in the art.
[0046] Some examples of 3D anatomical models include triangular meshes that represent the overall shape of anatomical structures (or their boundaries). Another example is a point cloud that identifies anatomical locations (for example, points representing cavities within anatomical structures or points representing elements that make up anatomical structures).
[0047] In step 130, one or more (2D) image sets 193 are also acquired. Each image set contains one or more 2D images of an anatomical structure from a specific anatomical viewpoint, so each image set provides a different anatomical viewpoint of the anatomical structure.
[0048] One or more image sets 193 are obtained by processing image data 191 and / or a 3D anatomical model 192. In one example, a classifier (e.g., a neural network or other machine learning method) is used to classify the 2D images of the image data to identify images that provide a specific anatomical viewpoint. In another example, the 3D anatomical model 192 is processed to extract one or more slices that provide different 2D images. In this way, an image set 193 is generated that has one or more images of different anatomical viewpoints.
[0049] Next, in step 140, one or more image sets 193 are processed to identify anatomical landmarks, "landmark points," or "key points" in each set of one or more 2D images. Anatomical landmarks are points of individual anatomical structures that show homology within an individual's species but may differ from person to person, such as the mitral valve point, the location of the fossa ovalis, or the location of the aortic valve.
[0050] For example, because it may be known which anatomical landmarks should be in a particular anatomical viewpoint, the anatomical landmarks identified in each set of one or more 2D images may differ depending on the anatomical viewpoint of the set of 2D images. Process 140 includes an image segmentation process to identify one or more predetermined anatomical landmarks within each 2D image. Approaches for identifying anatomical landmarks in 2D medical images are widely known and understood, and image segmentation processes, machine learning methods, automated classifiers, and landmark model adaptations can be employed.
[0051] Next, in step 150, the identified anatomical landmarks are processed to generate a 3D landmark model 194. Specifically, the identified landmarks are mapped to 3D coordinate space to generate a 3D landmark model. This is possible because the anatomical viewpoint of each image in the image set is already known (i.e., the relationship between the images in the image set and 3D space is already known). In this way, anatomical landmarks can be directly mapped or aligned to 3D coordinates, generating a patient-specific 3D landmark model with major anatomical features / junctions mapped in 3D.
[0052] Furthermore, the spatial relationship between the 3D anatomical model and the 3D landmark model can also be determined, as they are generated from the same image data. Therefore, the relative positions of anatomical landmarks relative to the 3D anatomical model obtained from the same image data can be easily determined. In other words, the proposed approach makes it possible to determine the relative position of specific anatomical landmarks within a 3D anatomical model.
[0053] Next, in process 160, the 3D anatomical model 192 is rendered and displayed. In this way, an image of the 3D anatomical model is generated for display to the user. The rendering of the 3D anatomical model is displayed, for example, on an output display or on a user interface such as a monitor or screen. The rendering and / or display of the 3D anatomical model is based on the 3D landmark model 194 and optionally on user input 197.
[0054] In particular, 3D landmark models are used to identify the locations of various anatomical features within a 3D anatomical model. This can then be used to improve and / or control the rendering of the 3D anatomical model, for example, by highlighting specific features of the rendering or by providing additional information about the anatomical features included in the rendering of the 3D anatomical model.
[0055] In some examples, one or more rendering parameters of the rendered 3D anatomical model are based on a 3D landmark model (and optionally user input). Rendering parameters include definitions of the rendering slice / section plane, the rendering camera viewpoint, position, and / or viewing direction, the rendering lighting conditions, and the rendering color and / or texture.
[0056] As mentioned earlier, a 3D landmark model defines the relative positions of specific anatomical landmarks to a 3D anatomical model. This information can be used to render the 3D anatomical model based on the positions of specific anatomical landmarks, for example, by highlighting or rendering desired anatomical features or views, highlighting specific anatomical features (e.g., using specific lighting parameters), or appropriately texturing specific anatomical features.
[0057] A fundamental understanding of the present invention is that by rendering and displaying 3D anatomical models using 3D landmark models, and / or providing additional information to 3D anatomical models rendered using 3D landmark models, it is possible to facilitate the provision of additional information to enhance the ease and certainty of evaluating anatomical features in the rendering of 3D anatomical models. This is achievable because 3D landmark models identify the location of specific anatomical landmarks within the 3D anatomical model and provide additional information to improve the rendering of the 3D anatomical model (or provide information about the rendered anatomical features of the anatomical model).
[0058] As mentioned above, the 3D anatomical model and the 3D landmark model are spatially aligned with each other; that is, they are formed in the same 3D coordinate space. This is possible because the 3D landmark model and the 3D anatomical model are generated from the same image data. In other words, the relationship between the 3D landmark model and the 3D anatomical model is maintained. This makes the rendering and display of the 3D anatomical model based on the 3D landmark model easier.
[0059] In some examples, one or more display parameters and / or displayed information are based on a 3D landmark model. For example, the anatomical viewpoint of a rendering is determined using a 3D landmark model and displayed (e.g., its text information) to facilitate user understanding. Another example is the use of a 3D landmark model to identify one or more anatomical features or parts of anatomical structures in a rendering.
[0060] A suitable example of how to render and display a 3D anatomical model based on a 3D landmark model is described below.
[0061] 3D rendering parameters are defined based on desired anatomical features or views to provide the user with a visualization of those desired anatomical features or views. Based on the 3D landmark model, the desired anatomical features or views of the 3D anatomical model are identified.
[0062] For example, one or more cross-sectional planes and / or viewing directions for rendering a 3D anatomical model can be defined (using a 3D landmark model) to provide the user with a rendering that includes and / or matches a desired anatomical view of the desired anatomical features. The anatomical features and / or anatomical view can be defined from the 3D landmark model (for example, by identifying anatomical landmarks associated with and / or present within the desired anatomical view of the desired features).
[0063] The cross-section is selected, as is well known in the art, to define a region / volume of a rendered 3D anatomical model and to include one or more desired anatomical features within the rendered region. The position and / or orientation of the cross-section is determined using a 3D landmark model to ensure that the cross-section is positioned so that the desired features are included in the rendered 3D anatomical model, for example, by identifying the relative position of the desired features (for example, based on a known positional relationship between the desired features and the landmarks).
[0064] The viewing direction defines the position and orientation of a virtual camera for rendering a 3D anatomical model, and is defined based on a 3D landmark model. For example, the viewing camera is defined to point towards a desired anatomical feature or view, and its position is calculated using the 3D anatomical model. In one scenario, a given viewing direction is known and includes a specific anatomical landmark at a specific relative position. Using the 3D landmark model, the appropriate position of the virtual camera (for rendering the 3D anatomical model) is selected to include the specific anatomical landmark at that particular position.
[0065] In this way, the 3D landmark model is processed to determine the appropriate position of the virtual camera to achieve the desired anatomical view and / or include the desired anatomical features in order to render the 3D anatomical model.
[0066] The rendering focus is defined as a desired anatomical feature or a central anatomical landmark for a desired anatomical view, and is also identified based on a 3D landmark model. For example, the focus for an overall visualization of the heart is defined as the cardiac cross.
[0067] The above example clearly illustrates embodiments in which the rendering content of the 3D anatomical model and the position of the virtual camera for rendering are based on a 3D landmark model (for example, to achieve a desired anatomical view and / or to render a desired anatomical feature).
[0068] The desired anatomical view and / or desired anatomical feature is indicated by user input 197 (if any). This input contains a specification for the desired anatomical feature or anatomical view. For example, the user selects the desired anatomical feature or anatomical view from a list of available anatomical features and / or views. For example, to image cardiac data, the user may desire a 4-chamber view, left ventricular outflow tract view, right ventricular outflow tract view, 3-vascular view, 3-vascular tracheal view, abdominal view, aortic arch view, or ductus arteriosus arch view. To image brain data, the user may desire a ventricular cross-sectional view, thalamic cross-sectional view, or cerebellar cross-sectional view.
[0069] In other cases, the desired anatomical view may be predefined or predetermined, such as by following a specific protocol to assess the patient's condition.
[0070] Another example of rendering parameters is that, for example, one or more lighting parameters for rendering a 3D anatomical model can be defined based on a 3D landmark model to effectively visualize desired anatomical features.
[0071] For example, during the rendering process, point lights can be placed to provide the user with an improved visualization of the rendered 3D representation. Spotlights or directional lights can be positioned and directed to illuminate or highlight specific / desired anatomical features. The position of the spotlight or directional light relative to the 3D anatomical model is determined based on the 3D landmark model.
[0072] As another example, backlighting can be used, for instance, across a valve to illuminate a specific range of anatomical features, providing the user with an optimal visualization of the valve-like anatomical structure. A 3D landmark model can be used to determine the position and / or direction of the backlighting, ensuring that specific anatomical features are illuminated in the rendered 3D representation.
[0073] Based on a 3D landmark model, different material properties or textures are applied to the rendering of a 3D anatomical model. Material properties are used to make the 3D representation (i.e., rendering) of anatomical structures look more realistic, making it easier for users to interpret the 3D representation and highlighting / emphasizing specific anatomical features. Examples of material properties include enhanced edge highlighting, gloss, and specular reflection highlighting, which, when combined, can produce various levels of polished appearance, rough / matte finishes, and surface translucency. The 3D landmark model can be used to determine which material properties to apply where.
[0074] In some examples, a 3D landmark model is used to locate the position of one or more anatomical features, for example, using the relationship between anatomical landmarks and anatomical features. The located anatomical features can then be textured (in rendering) to highlight, emphasize, or distinguish different anatomical features, or the anatomical features can be textured to more closely resemble their true texture.
[0075] For example, various types of rendering / texturing are used based on the content of the rendered 3D representation (which can be determined from the 3D landmark model). For instance, surface rendering is used to visualize the chambers of the heart, while translucent rendering is used to visualize the entire heart or the heart valves.
[0076] Various light transport models are used to improve the visualization of various anatomical features. Anatomical features are identified from 3D landmark models. For example, when visualizing heart valves, shadows are undesirable because they can obscure dark artifacts (i.e., rendering may omit the rendering of shadows). On the other hand, shadows improve the visualization of the entire heart or cardiac chambers by providing shading and depth cues, allowing users to distinguish structures more easily.
[0077] The above examples illustrate embodiments in which one or more rendering parameters of the 3D representation depend on the 3D landmark model. In some examples, one or more rendering parameters are defined by user input 197, and other rendering parameters are defined by the 3D landmark model.
[0078] For example, user input 197 provides instructions for desired rendering parameters of the rendered 3D representation of a 3D anatomical model, such as the position of the cross-section and / or virtual camera (e.g., the viewing direction of the rendering). The 3D landmark model is used to identify or predict the anatomical features and / or anatomical views provided by the desired rendering. Based on the predicted anatomical features and / or anatomical views, the 3D landmark model is used to define further rendering parameters such as lighting conditions, color, and / or texture.
[0079] All of the above examples illustrate embodiments in which one or more rendering parameters of a 3D representation of an anatomical structure are based on a 3D landmark model. In some examples, the 3D landmark model is not used for rendering but is used to generate additional information (e.g., in the form of text information) about the 3D representation of the anatomical structure.
[0080] For example, rendering parameters for the 3D representation of anatomical structures may be predefined and / or responsive to user input. Information about the 3D representation provided to the user is generated using a 3D landmark model.
[0081] For example, rendering parameters and a 3D landmark model are processed to identify anatomical features included in the 3D representation, or the (closest) anatomical view provided by the 3D representation. This information is then displayed to the user in the form of text output or graphic output showing a relative view to the anatomical model.
[0082] For example, additional information is generated and optionally displayed, such as information indicating the suitability of the anatomical view, the direction in which the 3D anatomical model needs to be rotated / re-sliced to snap to one or more of the nearest available standard views, and the direction in which it needs to be crossed.
[0083] Therefore, supporting information to assist the user in controlling and understanding the input to achieve the desired anatomical view is provided by processing the current parameters of the 3D representation and 3D landmark model.
[0084] In some embodiments, the anatomical structure exhibits periodic motion. For example, the anatomical structure includes the heart or lungs. The image data 191 includes image data captured over at least one entire period of periodic motion, such as over at least one cardiac cycle or at least one respiratory cycle. In this case, each of one or more image sets 193 includes a sequence of 2D images representing at least one entire period of periodic motion. One or more predetermined anatomical landmarks are identified within each image in each image set, and thus the anatomical landmarks are tracked over the entire period.
[0085] In this way, the 3D landmark model effectively tracks the positions of anatomical landmarks in the 3D anatomical model over time. This information can be used to control the rendering of the 3D representation of the anatomical structure represented by the 3D anatomical model, for example, to render the 3D representation at a specific point in time within a period.
[0086] Figure 2 shows a set of 2D images of the fetal heart obtained from ultrasound image data over one cardiac cycle. This set represents four chamber views of the fetal heart at different phases of the cardiac cycle, with 12 anatomical landmarks tracked across the phases. Points 201 and 202 are the midpoint between the mitral and tricuspid valves, point 203 is the cardiac cross, points 204 and 205 are the endpoints of the mitral and tricuspid valves, point 206 is the vertebra, point 207 is the aorta, point 208 is the apex, points 209 and 210 are the left and right atria, and points 211 and 212 are the left and right ventricles.
[0087] Further sets of 2D images can be obtained from the same ultrasound imaging data, each representing a different predetermined view. For example, further sets of 2D images corresponding to left ventricular outflow tract views, right ventricular outflow tract views, three-vascular views, three-vascular tracheal views, abdominal views, aortic arch views, and / or ductus arteriosus arch views can be obtained from the ultrasound image data. Anatomical landmarks visible in each set of 2D images are tracked throughout all phases of the cardiac cycle. The tracked points from all views are mapped to 3D coordinates to generate a 3D landmark model.
[0088] Figure 3 shows views of the left ventricle of the heart in a four-chamber view with different rendering parameters applied. The viewing parameters for the views shown in Figure 3 are adjusted or defined using a 3D landmark model, for example, to improve visualization or to ensure that the desired view is achieved.
[0089] The 3D rendering cross-sections are positioned on both sides of the arterial and ventricular valves. The viewing direction is defined with respect to the position, orientation, and focus of the virtual camera. The camera's view-up vector is directed from the left atrium to a key point in the left ventricle. The projection direction is orthogonal to the view-up vector, and the axial plane is defined by the center of the ventricle. The focus is the center of the left ventricle. The camera is positioned to achieve a field of view that is twice the distance between the left and right ventricles.
[0090] In this example, the viewing parameters (i.e., the cross-sectional plane and viewing direction) are defined using the locations of anatomical features. These locations are determined based on a 3D landmark model generated by processing the anatomical landmarks identified as described above. Therefore, the rendering of the 3D representation is based on the 3D landmark model.
[0091] Images 310, 320, and 330 show views of the left ventricle rendered with different lighting parameters. In Image 310, the left ventricle is illuminated by ambient light. In Images 320 and 330, a spotlight is positioned in the left ventricle, with the direction of the spotlight perpendicular to the axial plane and directed towards key points in the left ventricle. In Image 320, the spotlight's cutoff is set to one-quarter of the distance between the left and right ventricles, as indicated by identified anatomical landmarks. In Image 330, the cutoff is set to half the distance between the left and right ventricles.
[0092] The position and characteristics of the light source are also determined based on the 3D landmark model.
[0093] Images 340, 350, and 360 show views of the left ventricle rendered with different material properties. Image 370 shows a view of the left ventricle with a spherical shadow applied. Image 380 shows a view of the left ventricle with a surface rendering applied.
[0094] To visualize anatomical structures exhibiting periodic motion, such as the heart, a 3D representation of the anatomical structure at a desired point in the cycle can be displayed by tracking anatomical landmarks identified throughout the entire cycle. Therefore, the 3D representation of the anatomical structure represents the structure at a specific point in the cycle. This specific point in time is detected by tracking the movement of anatomical landmarks using a 3D landmark model.
[0095] In other words, a 3D anatomical model tracks the positions of anatomical landmarks in 3D coordinate space over a predetermined period. Similarly, the 3D model also changes over the same predetermined period. Rendering of the 3D anatomical model is performed to render the 3D anatomical model at a specific point in time within the period. This specific point in time is selected based on the 3D anatomical model.
[0096] In some examples, one or more rendering parameters of a 3D representation are defined or modified based on a point in time within a period. For example, the position of a spotlight illuminating an anatomical feature changes as the position of the anatomical feature changes during the period (for example, as it is tracked by a 3D landmark model).
[0097] Figure 4 shows views of the mitral and tricuspid valves from valve opening (image 410) to valve closure (image 430). Using tracking points 201 and 202, which correspond to the midpoints of the mitral and tricuspid valves shown in Figure 2, you can select the view frame at the appropriate time point within the cardiac cycle.
[0098] In this way, the rendered 3D representation can provide a 3D representation of the desired anatomical structure at a specific point in time within the anatomical structure's cycle. This approach can provide more useful images for assessing a patient's condition, for example, by automatically "freezing" the rendering of the 3D anatomical structure at a clinically useful point in time within the anatomical structure's cycle.
[0099] An induced viewing workflow can be created using the proposed approach of rendering 3D representations with 3D landmark models and 3D anatomical models. This induced viewing workflow is designed to comply with clinical guidelines, for example, to evaluate or diagnose an individual.
[0100] For example, the guided viewing workflow conforms to the ISUOG guidelines for fetal ultrasound scanning. In guided fetal cardiac examination, a 3D visualization of the four-chamber view of the fetal heart is displayed first, followed by 3D visualizations of the left ventricular outflow tract view, right ventricular outflow view, three-vascular view, three-vascular tracheal view, and then examination of valves, septa, etc. Using the proposed approach with 3D landmark models and 3D anatomical models, appropriate rendering parameters (using the 3D landmark model) can be appropriately defined for rendering the 3D anatomical model to achieve each view.
[0101] In some cases, visualizations of each view defined by a set of guidelines can be automatically bookmarked and saved, allowing for searchable views of each structure. Bookmarked visualizations are generated using appropriate rendering parameters (section plane, viewing parameters, lighting parameters, etc.) to optimize the display of the structure evaluated in that view, as described above. In this way, an optimal set of visualizations corresponding to the visualizations required by the set of guidelines is automatically generated. In this case, the desired anatomical feature or view from the bookmarked visualizations is displayed simply by the user's selection.
[0102] Using 3D landmark models allows for the automation of the interaction between the displayed 3D representation and multiplanar reformation (MPR). Multiplanar reformation requires careful adjustment of the MPR plane to reach the desired plane. Traditionally, the operator had to manipulate the x, y, and z knobs and the translation knob to translate and rotate the plane and adjust its pivot point. This is cumbersome and requires a skilled operator.
[0103] MPR interactions are automated by positioning pivot points (in the 3D anatomical model) based on identified anatomical landmarks within the 3D landmark model. In this way, the pivot points are automatically optimized for specific views. For example, interactions for a 4-chamber view of the heart are best performed on an MPR plane centered on the cardiac cross. Interactions for a 3-vascular view or a 3-vascular tracheal view are best performed on an MPR plane centered on the aorta. The extent of the MPR plane is automatically defined for each view by determining the cross-sectional plane that defines the limits of each view using the 3D landmark model. For anatomical structures exhibiting periodic motion, the view is displayed at the most appropriate point in time within the telophysiological phase.
[0104] For example, using the 3D anatomical model and 3D landmark model generated as described above, a set of visible planes is created. A list of visible planes is displayed to the user, and the user selects the desired plane. Interaction constraints on the selected plane are automatically applied by defining the center of rotation of the plane, the range of translation and rotation of the cross-section, so that any view obtained during MPR interaction corresponds to the selected view. For anatomical structures exhibiting periodic motion, the automatic constraints may be temporary in order to enable the display of dynamic MPR. For example, the display may be limited to one period of the anatomical structure.
[0105] 3D landmark models are used in MPR-assisted tools. For example, users can freely interact with the displayed 3D representation, rotate and adjust the cutting plane, and see additional information along with the display of re-sliced images. This additional information includes, for example, the type of view being displayed, indications of view suitability, the plane range of the displayed view, and directional markers that guide the user to one or more of the nearest available standard views. The additional information is determined based on the 3D landmark model. For example, the type of view being displayed is determined based on which anatomical landmarks are visible and their locations.
[0106] In another example, a 3D landmark model is used to generate a personalized model that is overlaid on the cut view during live reslicing. The personalized model is generated, for example, by deforming the ideal model using a thin plate spline warping transformation, with the key points of the ideal model as source landmarks and the anatomical landmarks of the 3D landmark model as target landmarks. The personalized model may be overlaid on the cut view using any suitable augmented reality technique. For example, the personalized model may be overlaid as a surface rendering.
[0107] Figure 5 shows a system 500 including a processing system 510 according to an embodiment of the present invention, a memory 520, and an output display 530. The processing system 510 generates and displays a 3D representation of an individual's anatomical structure.
[0108] The processing system 510 acquires image data of anatomical structures. The image data may be acquired, for example, from memory 520, or directly from an image data generator (for example, an imaging system such as an ultrasound scanner).
[0109] The processing system also processes image data to generate a 3D anatomical model of the anatomical structure, and processes the image data and / or the 3D anatomical model to obtain one or more sets of one or more 2D images. Each identified set of one or more 2D images represents a different predetermined view of the anatomical structure.
[0110] The processing system 510 also identifies one or more predetermined anatomical landmarks within each set of one or more 2D images, and processes one or more predetermined anatomical landmarks to generate a 3D landmark model.
[0111] The processing system 510 also processes 3D anatomical models and 3D landmark models and renders and displays 3D representations of anatomical structures on the output display 530.
[0112] For example, the output display 530 includes a monitor or screen that provides a visual output of a 3D representation of an anatomical structure provided by the processing system 510.
[0113] Those skilled in the art will be able to easily develop a processing system for carrying out the method described herein. Thus, each step in the flowchart represents a different action to be performed by the processing system, which can be performed by the corresponding module of the processing system.
[0114] Processing systems can be implemented in various ways using software and hardware to perform a variety of necessary functions. A (micro)processor is an example of a processing system that employs one or more microprocessors programmed using software (e.g., microcode) to perform the necessary functions. However, processing systems can be implemented regardless of the use of processors, and can also be implemented as a combination of dedicated hardware for performing some functions and processors (e.g., one or more programmed microprocessors and associated circuits) for performing other functions.
[0115] Modifications of the disclosed embodiments can be understood and implemented by those skilled in the art in carrying out the claimed invention, based on a review of the drawings, disclosures, and appended claims. In the claims, the word “includes” does not exclude other elements or steps, and singular elements do not exclude plural elements. The mere fact that certain means are described in different dependent claims does not mean that combinations of these means cannot be used advantageously.
[0116] A single processor or other unit may perform the functions of some of the items described in the claims. Computer programs may be stored / distributed on any suitable medium, such as optical storage media or solid-state media, supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless communication systems.
[0117] Note that when the term "adapted to..." is used in the claims or description, it is intended to be equivalent to the term "configured to...". Any reference numerals in the claims should not be construed as limiting the scope.
Claims
1. 1. A computer-implemented method for generating and displaying a 3D representation of an individual's anatomy, the computer-implemented method comprising: acquiring image data of the anatomical structure; processing the image data to generate a 3D anatomical model of the anatomical structure; processing the image data and / or the 3D anatomical model to obtain one or more sets of one or more 2D images, each obtained set of one or more 2D images representing a different predefined view of the anatomical structure; processing each acquired set of one or more 2D images to identify one or more predefined anatomical landmarks within each set of one or more 2D images; processing the one or more predefined anatomical landmarks to generate a 3D landmark model; processing the 3D anatomical model and the 3D landmark model to render and display the 3D anatomical model to generate a 3D representation of the anatomical structure on an output display; Including, A computer-implemented method, wherein processing the 3D anatomical model and the 3D landmark model includes defining at least one rendering parameter based on the 3D landmark model, the at least one rendering parameter including one or more of: a cutting plane of the rendered 3D representation and / or a viewing direction of the rendered 3D representation.
2. the image data includes image data of the anatomical structure captured over at least one period of periodic motion of the anatomical structure; each set of one or more 2D images comprising a sequence of 2D images representing at least one period of the periodic motion of the anatomical structure; The step of processing the one or more predetermined anatomical landmarks further comprises: tracking one or more predetermined landmarks in each sequence of 2D images; generating the 3D landmark model based on the one or more predetermined landmarks tracked in each sequence of 2D images; The computer-implemented method of claim 1 , comprising:
3. The computer-implemented method of claim 1 or 2, wherein the at least one rendering parameter includes one or more of: lighting parameters of the rendered 3D representation, one or more textures of the rendered 3D representation, and / or one or more colors of the rendered 3D representation.
4. 4. The computer-implemented method of claim 1, wherein processing the one or more predetermined anatomical landmarks comprises mapping the one or more predetermined landmarks in each set of one or more 2D images to a 3D coordinate system to generate a 3D landmark model.
5. Rendering and displaying the 3D anatomical model comprises: registering the 3D landmark model and the 3D representation in the same 3D coordinate space; controlling the rendering of the 3D anatomical model based on the 3D landmark model of the anatomical structure; 5. The computer-implemented method of claim 1 , comprising:
6. Rendering and displaying the 3D anatomical model comprises: defining a viewing orientation for the rendering of the 3D anatomical model based on the 3D landmark model and a desired anatomical feature and / or anatomical view; rendering and displaying the 3D anatomical model to generate the 3D representation of the anatomical structure based on the defined viewing direction; 6. The computer-implemented method of claim 1 , comprising:
7. Rendering and displaying the 3D anatomical model comprises: defining one or more cutting planes of the rendering of the 3D anatomical model based on the 3D landmark model and desired anatomical features and / or anatomical views; rendering and displaying the 3D anatomical model to generate the 3D representation of the anatomical structure based on the one or more defined cutting planes; 7. The computer-implemented method of claim 1, comprising:
8. The computer-implemented method of claim 6 or 7, further comprising receiving, at a user interface, a first user input indicating the desired anatomical feature or anatomical view.
9. The computer-implemented method of claim 6 , wherein the desired anatomical feature and / or anatomical view is one of a plurality of predefined anatomical views of the anatomical structure.
10. The computer-implemented method of claim 1 , wherein the anatomical structure is a heart.
11. 11. The computer-implemented method of claim 10, wherein each identified set of one or more 2D images represents a different one of a four-chamber view, a left ventricular outflow tract view, a right ventricular outflow tract view, a three-vessel view, a three-vessel tracheal view, an abdominal view, an aortic arch view, and / or an arterial arch view.
12. 12. The computer-implemented method of claim 1, wherein processing the image data and / or the 3D anatomical model comprises processing the image data and / or the 3D anatomical model using machine learning methods to identify one or more sets of one or more 2D images.
13. 13. A computer program comprising computer program code means which, when executed on a computing device having a processing system, causes the processing system to perform all the steps of the method according to any one of claims 1 to 12.
14. 1. A processing system for generating and displaying a 3D representation of an individual's anatomy, the processing system comprising: acquiring image data of the anatomical structure; processing the image data to generate a 3D anatomical model of the anatomical structure; processing the image data and / or the 3D anatomical model to obtain one or more sets of one or more 2D images, each obtained set of one or more 2D images representing a different predefined view of the anatomical structure; processing each acquired set of one or more 2D images to identify one or more predefined anatomical landmarks within each set of one or more 2D images; processing the one or more predetermined anatomical landmarks to generate a 3D landmark model; processing the 3D anatomical model and the 3D landmark model to render and display the 3D anatomical model to generate a 3D representation of the anatomical structure on an output display; Run The processing system processes the 3D anatomical model and the 3D landmark model using a process that includes defining at least one rendering parameter based on the 3D landmark model, the at least one rendering parameter including one or more of: a cutting plane of the rendered 3D representation and / or a viewing direction of the rendered 3D representation.
