A semiconductor device and a method of fabricating the same are disclosed. The semiconductor device includes a substrate, a shallow trench isolation, a gate structure, a residue, and a spacer. The shallow trench isolation is disposed in the substrate and includes an inclined side surface at a side close to the substrate. The gate structure is disposed on the shallow trench isolation. The residue is disposed and physically contacts the inclined side surface of the shallow trench isolation. The spacer is disposed on a side wall of the gate structure and physically contacts the side wall of the gate structure, the shallow trench isolation, and the residue. Thus, the residue fills up a small recess or unevenness on a surface of the shallow trench isolation, thereby optimizing a component structure and performance of the semiconductor device.