A method and system for patient-specific modeling of the whole heart anatomy, dynamics, hemodynamics, and fluid structure interaction from 4D medical image data is disclosed. The anatomy and dynamics of the heart are determined by estimating patient-specific parameters of a physiological model of the heart from the 4D medical image data for a patient. The patient-specific anatomy and dynamics are used as input to a 3D Navier-Stokes solver that derives realistic hemodynamics, constrained by the local anatomy, along the entire heart cycle. Fluid structure interactions are determined iteratively over the heart cycle by simulating the blood flow at a given time step and calculating the deformation of the heart structure based on the simulated blood flow, such that the deformation of the heart structure is used in the simulation of the blood flow at the next time step. The comprehensive patient-specific model of the heart representing anatomy, dynamics, hemodynamics, and fluid structure interaction can be used for non-invasive assessment and diagnosis of the heart, as well as virtual therapy planning and cardiovascular disease management. Parameters of the comprehensive patient-specific model are changed or perturbed to simulate various conditions or treatment options, and then the patient specific model is recalculated to predict the effect of the conditions or treatment options.