The present invention relates to a method of forming an isolation film of a semiconductor device. According to the present invention, the method includes the steps of forming a pad film on a semiconductor substrate, and patterning the pad film of a predetermined region and a predetermined depth of the semiconductor substrate to form trenches, forming sidewall oxide films on sidewalls of the trenches thereby defining a non-active region and an active region, forming a first oxide film for trench burial on the entire surface including the sidewall oxide films, and then performing a polishing process until the pad film is exposed, thus forming a first isolation film, removing the pad film to expose the semiconductor substrate in the active region, forming a silicon layer, which has a height higher than that of the first isolation film, on the exposed semiconductor substrate in the active region, and forming a second oxide film for trench burial on the entire surface, and then performing a polishing process until the silicon layer is exposed, thus forming a second isolation film, whereby an isolation film in which the first isolation film and the second isolation film are stacked is formed.