The invention relates to a method and device of generating virtual surround sound. The method comprises environment sound and audio frequency object treatment flows. The environment sound treatment flow comprises first order B format signal input, spinning by a spin matrix, binaural transcoding based on a HRTF (Head-Related Transfer Function), and binaural output. The audio frequency object treatment flow comprises encoding one or more than one audio frequency object into a higher-order ambisonic signal, performing binaural transcoding based on BRIR (Binaural Room Impulse Response), performing binaural output, and respectively outputting environment sound and an audio frequency object after corresponding mixture according to left and right sound channels. The device comprises a head tracking means, an audio frequency object input module, an audio frequency object B format coding module, an audio frequency object virtual loudspeaker array signal generation module and an audio frequency object BRIR module. The method and device can effectively generate virtual surround sound with high quality, and are mainly used for cooperating with a virtual reality HMD (head-mounted display) to perform audio frequency stereo headphone playing.