The invention discloses a handwritten letter recognizing method and system based on WiFi. The method includes the following steps of A1, collecting WiFi signals for reflecting environment state changes through a data collecting module when a user writes handwritten letters between an AP and an STP; A2, extracting channel state information from the WiFi signals; A3, conducting phase unwinding and phase correcting operations on the channel state information; A4, inputting data into a convolutional neural network, wherein the convolutional neural network comprises an input layer, an Inception module, a depth connecting layer, a batch standardization layer, a ReLU layer, an average pooling layer, a dropout layer, a full-connection layer, a softmax layer and a classification layer, and the network is trained through a momentum gradient descending method; A5, applying a trained convolutional neural network model to handwritten letter recognizing. The limitation that a wearable device needs to be carried during traditional action recognition and the risk of privacy leakage are overcome. Action recognition is conducted by means of the improved convolutional neural network, and the recognizing precision is improved.