Provided are a nonvolatile semiconductor memory device capable of improving data retention characteristics and a method of manufacturing the same. On the upper layer of the memory element, there is at least one layer including at least one of the following groups: a silicon oxide film doped with nitrogen; a silicon oxide film doped with Al; an oxide of Al; a silicon oxide film doped with Ti; Nitrogen and two of the three kinds of Al and Ti silicon oxide film; silicon oxide film with nitrogen and three kinds of Al and Ti added; Ti oxide; Ti and Al oxide; made of Ti, Ni, Co , Zr, Cu, Pt, V, Mg, U, Nd, La, Sc metal group consisting of a single metal layer; metals containing two or more of these metal groups account for at least 50% of the whole A layer composed of a binary or higher alloy; a layer composed of a nitride of the alloy; or a layer composed of a hydride of the alloy (such as the Al2O3 film 10).