Method and system for identifying footstep sound based on deep neural network

Abstract

The application provides a method for identity recognition of footstep sound based on a deep neural network, comprising the following steps: S1, collecting footstep sound frequency information to form a database, and dividing footstep events of data in the database; S2, further dividing the footstep events, extracting features of waveforms, manual beat maps and mel spectra in the footstep events, and establishing a double-footstep event sample library; S3, inputting the features of the double-footstep event sample library into a CRNN deep neural network for further extraction, and standardizing and fusing the extracted features to obtain fused features; S4, training an identification model by using the fused features; and S5, inputting to-be-detected footstep sound frequency information into the identification model, and performing identity recognition by matching the to-be-detected footstep sound frequency information with information in the identification model. The application pays more attention to privacy protection in recognizing footstep information, and has better applicability in different environmental conditions such as different illuminations.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, in particular to an identity recognition method based on footstep sound frequency, and more particularly to a method and system for identity recognition of footstep sound based on deep neural network. BACKGROUND

[0002] With the continuous progress of technology, identity recognition technology has become an indispensable part of modern society. It plays an important role in finance, security, medical care and other fields. Although the traditional password or physical key-based identity verification method is simple, it has the risk of being forgotten, lost or cracked. Therefore, biometric technology has gradually become a research hotspot due to its uniqueness and difficulty in replication. Biometric technology includes fingerprint recognition, facial recognition, iris recognition, voiceprint recognition, vein recognition, body posture recognition, gait recognition, etc., which use individual physiological or behavioral characteristics to achieve identity verification.

[0003] Among many biometric technologies, identity recognition technology based on footstep sound has attracted attention due to its unique advantages. Compared with visual-based methods, it does not require users to face the camera directly, thereby reducing the risk of privacy leakage. In addition, footstep sound recognition technology is not limited by lighting conditions and can be used in dark environments, and the requirements for hardware devices are relatively low. These characteristics make the identity recognition technology based on footstep sound have broad application prospects in the field of access control based on footstep sound for stranger recognition.

[0004] A search revealed only two recent patents related to footstep audio-based identity recognition methods. Publication number CN200810199168.4, titled "An Intelligent Access Control System Based on Footstep Recognition," also uses footstep audio as a data source. This footstep recognition and processing method primarily employs traditional signal processing techniques and pattern recognition methods, such as endpoint detection, filtering, LPC, LPC cepstrum, and template matching. These methods rely on classical theories of signal processing and pattern recognition to identify footsteps. Compared to these methods, our approach extracts various features from footstep audio, such as beat patterns and Mel-frequency cepstrum coefficients, constructs a CRNN model, and employs a deep learning neural network approach for identity recognition. Another patent, published with the publication number CN201711349393.7 and titled "A Method for Identification Based on Multimodal Perception," establishes a feature database by collecting wireless channel state information (CSI) and microphone sound signals. By comparing the features extracted during the identification phase with those extracted during the training phase, a support vector machine (SVM) classifier is applied to the multimodal features. The classification results of the multiple classifiers are then integrated to achieve user identification. Compared to this method, our method, based solely on footstep audio signals, extracts more features from the footstep audio itself and employs a more modern deep learning neural network approach for identification.

[0005] To address the issues existing in previous work and existing methods, we proposed a method for footstep identification based on deep neural networks (AFPI-ATM). This method uses footstep audio as the data source, which simplifies the acquisition process and is less affected by environmental conditions such as lighting. To enable the model to more effectively extract gait features, we extracted three branch audio features that can simultaneously reflect the overall and local characteristics of the audio. Furthermore, we designed a deep learning-based feature extraction network to better extract features for identification. Summary of the Invention

[0006] In view of the defects in the prior art, the purpose of the present invention is to provide a method and system for identifying the identity of footsteps sounds based on deep neural networks.

[0007] According to the present invention, a method for identifying footstep sounds based on a deep neural network is provided, comprising:

[0008] Step S1: collecting footstep audio information to form a database, and dividing the database data into footstep events;

[0009] Step S2: further divide the footstep events, extract the features of the waveform, manual beat graph, and Mel spectrum in the footstep events, and establish a double footstep event sample library;

[0010] Step S3: input the features of the double-foot step event sample library into the CRNN deep neural network for further extraction, and standardize and fuse the extracted features to obtain fused features;

[0011] Step S4: train the fused features to recognize the model;

[0012] Step S5: input the to-be-tested footstep sound frequency information into the recognition model, and perform identity recognition by matching with the information in the recognition model.

[0013] Preferably, the step S1 comprises:

[0014] Step S1.1: collect footstep sound frequency information within a preset range of the microphone, and establish a database;

[0015] Step S1.2: search for local maximum values of the footstep sound frequency every first time interval;

[0016] Step S1.3: divide the footstep events; take the time at which the local maximum value of the audio is located as the starting point, offset the starting time direction and the ending time direction by a second preset time interval respectively, and take the two end points after the offset as the starting point and the ending point of the footstep sound frequency time. The audio information from the starting point to the ending point is a footstep event;

[0017] Wherein, the second time interval is less than the first time interval, and the first time interval is the minimum horizontal distance between adjacent peak values;

[0018] Repeat steps S1.2-S1.3 until all audio information in the database is divided into footstep events, and enter step S2.

[0019] Preferably, the step S2 comprises:

[0020] Step S2.1: according to the divided audio, take two adjacent footstep events as a single double-foot sample;

[0021] Step S2.2: extract waveform features in the single double-foot sample; specifically, extract the audio waveform in the single double-foot sample as the first feature, and the first feature is a 64x1 audio amplitude vector;

[0022] Step S2.3: extract the hand-tapped beat map feature in the single double-foot sample, and input the hand-tapped beat map feature into the beat map function to obtain a 64x70 feature matrix for representing the second feature;

[0023] The hand-tapped beat map feature includes root mean square feature, zero-crossing rate feature, spectral center feature, and spectral flatness feature;

[0024] Step S2.4: In a single two-foot sample, extract the mel-spectrogram feature, and obtain a 64x70 third feature by calculating the mel-frequency cepstral coefficient;

[0025] Step S2.5: Repeat steps S2.1-S2.4 until all the features of the single two-foot samples are extracted, and a two-foot step event sample library is established, and all the extracted features are put into the two-foot step event sample library.

[0026] Preferably, the step S3 comprises:

[0027] Step S3.1: The features of the two-foot step event sample library are respectively input into the three branch structures of the CRNN deep neural network for further feature extraction;

[0028] The first branch structure adopts a filter block as the first layer, each block is composed of a convolution layer, a batch normalization layer and a maximum pooling layer, and is used for down-sampling; the second branch structure is composed of two layers of GRU units, and is bidirectional calculation; the third branch structure is composed of four sub-blocks, each block is composed of a two-dimensional convolution layer, a batch normalization layer and a maximum pooling layer;

[0029] Step S3.2: The extracted features are standardized and fused to obtain a fused feature;

[0030] Specifically, the end of each of the three branches adopts an adaptive average pooling layer, and the same length of output features are obtained through fusion, and the output features are combined in an element-wise summation manner to obtain the fused feature.

[0031] Preferably, the step S4 comprises:

[0032] Step S4.1: The fused feature is input into a classification network for identity labeling;

[0033] Step S4.2: An identification model is established, and the labeled fused feature is input into the identification model for model training.

[0034] According to the system for identifying the identity of the footstep sound based on the deep neural network provided by the application, the footstep sound frequency information is collected to form a database, and the data of the database is divided into footstep events;

[0035] Module M1: Collecting footstep sound frequency information to form a database, and dividing the data of the database into footstep events;

[0036] Module M2: Further dividing the footstep events, extracting the features of the waveform, the manual beat figure and the mel-spectrum in the footstep events, and establishing a two-foot step event sample library;

[0037] Module M3: input the features of the double-footstep event sample library into the CRNN deep neural network for further extraction, and standardize and fuse the extracted features to obtain fused features;

[0038] Module M4: train the fused features to identify the model;

[0039] Module M5: input the to-be-tested footstep audio information into the identification model, and perform identity recognition by matching with the information in the identification model.

[0040] Preferably, the module M1 comprises:

[0041] Module M1.1: collect footstep audio information within a preset range of the microphone, and establish a database;

[0042] Module M1.2: search for local maximum values of footstep audio every first time interval;

[0043] Module M1.3: divide footstep events; take the time at which the local maximum value of the audio is located as the starting point, offset the starting time direction and the ending time direction by a second preset time interval respectively, and take the two end points after the offset as the starting point and the ending point of the footstep audio time. The audio information from the starting point to the ending point is a footstep event;

[0044] Wherein, the second time interval is less than the first time interval, and the first time interval is the minimum horizontal distance between adjacent peak values;

[0045] Repeat the module M1.2-module M1.3 until all audio information in the database is divided into footstep events, and enter the module M2.

[0046] Preferably, the module M2 comprises:

[0047] Module M2.1: take two adjacent footstep events as a single double-footstep sample according to the divided audio;

[0048] Module M2.2: extract waveform features in the single double-footstep sample; specifically, extract the audio waveform in the single double-footstep sample as the first feature, and the first feature is a 64x1 audio amplitude vector;

[0049] Module M2.3: extract hand-tapped beat map features in the single double-footstep sample, and input the hand-tapped beat map features into a beat map function to obtain a 64x70 feature matrix for representing the second feature;

[0050] The hand-tapped beat map features include root mean square features, zero-crossing rate features, spectral center features, and spectral flatness features;

[0051] Module M2.4: In a single two-foot sample, the mel-spectrogram feature is extracted, and the third feature of 64*70 is obtained by calculating the mel-frequency cepstral coefficient;

[0052] Module M2.5: Repeat the triggering module M2.1-M2.4 until all the features of the single two-foot sample are extracted, and a two-foot step event sample library is established, and all the extracted features are put into the two-foot step event sample library.

[0053] Preferably, the module M3 comprises:

[0054] Module M3.1: The features of the two-foot step event sample library are input into the three branch structures of the CRNN deep neural network respectively to further extract the features;

[0055] Among them, the first branch structure adopts filter blocks as the first layer, each block is composed of a convolution layer, a batch normalization layer and a maximum pooling layer, and is used for down sampling; the second branch structure is composed of two layers of GRU units, and is calculated bidirectionally; the third branch structure is composed of four sub-blocks, each block is composed of a two-dimensional convolution layer, a batch normalization layer and a maximum pooling layer;

[0056] Module M3.2: The extracted features are standardized and fused to obtain the fused features;

[0057] Specifically, the end of the three branches adopts an adaptive average pooling layer, and the same length of output features is obtained through fusion, and the output features are combined in an element-by-element summation manner to obtain the fused features.

[0058] Preferably, the module M4 comprises:

[0059] Module M4.1: The fused features are input into a classification network for identity labeling;

[0060] Module M4.2: An identification model is established, and the labeled fused features are input into the identification model for model training.

[0061] Compared with the prior art, the present application has the following beneficial effects:

[0062] 1. Compared with the traditional camera-based method, the present application pays more attention to privacy protection in recognizing step information, and has better applicability in different environmental conditions such as light;

[0063] 2. The application first proposes to add a beatmap feature as a new feature to the second feature of the three features extracted from the input audio, effectively extracting the beatmap feature from the audio signal, increasing the recognition of the foot rhythm, since the foot rhythm feature of each identity is unique, the expression ability of the audio feature is improved, the characteristics of the audio signal can be more comprehensively described, and the accuracy of the audio recognition is enhanced;

[0064] 3. According to the new feature extraction method, a new foot sound feature extraction network is designed based on the space-time convolution network; the network for foot sound is used for feature extraction, and the learned features have the advantages of strong expression ability and generalization. BRIEF DESCRIPTION OF DRAWINGS

[0065] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments with reference to the following drawings:

[0066] Figure 1 The flowchart of the method for identifying the identity of the foot sound based on the deep neural network of the present application is shown in the figure;

[0067] Figure 2 The flowchart of the method of embodiment 1 of the present application is shown in the figure;

[0068] Figure 3 The framework diagram of the method of embodiment 1 of the present application is shown in the figure;

[0069] Figure 4 The schematic diagram of the deep neural network (AFPI-ATM) of the present application is shown in the figure;

[0070] Figure 5 The flowchart of the recognition model training of the present application is shown in the figure. DETAILED DESCRIPTION

[0071] The present application will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be pointed out that those skilled in the art can make several changes and improvements without departing from the concept of the present application. These all belong to the protection scope of the present application.

[0072] The present application belongs to the field of computer vision, and specifically relates to a method for identifying the identity based on foot sound frequency. It also belongs to the field of artificial intelligence, and specifically relates to feature extraction, fusion and model training based on deep neural network. The present application belongs to the field of pattern recognition, and specifically designs to identify the identity of foot sound.

[0073] A method for identifying the identity of foot sound based on deep neural network, comprising:

[0074] Step S1: Collecting footstep audio information to form a database, and dividing the data of the database into footstep events;

[0075] Step S2: Further dividing the footstep events, extracting the waveform, manual beat figure and mel-frequency spectrum features in the footstep events, and establishing a double-footstep event sample library;

[0076] Step S3: Inputting the features of the double-footstep event sample library into a CRNN deep neural network for further extraction, and standardizing and fusing the extracted features to obtain fused features;

[0077] Step S4: Training the fused features to recognize a model;

[0078] Step S5: Inputting the to-be-tested footstep audio information into the recognition model to perform identity recognition by matching with the information in the recognition model.

[0079] Specifically, the step S1 includes:

[0080] Step S1.1: Collecting footstep audio information within a preset range of a microphone, and establishing a database;

[0081] Step S1.2: Searching for a local maximum value of the footstep audio every first time interval;

[0082] Step S1.3: Dividing the footstep events; taking the time at which the local maximum value of the audio is located as a starting point, respectively offsetting a second preset time interval in the starting time direction and the ending time direction, and taking the two offset end points as the starting point and the ending point of the footstep audio time, the audio information from the starting point to the ending point being a footstep event;

[0083] The second time interval is less than the first time interval, and the first time interval is the minimum horizontal distance between adjacent peak values.

[0084] Repeat steps S1.2-S1.3 until all audio information in the database is divided into footstep events, and proceed to step S2.

[0085] Specifically, the step S2 includes:

[0086] Step S2.1: Taking two adjacent footstep events as a single double-foot sample according to the divided audio;

[0087] Step S2.2: Extracting waveform features in the single double-foot sample; specifically, extracting an audio waveform in the single double-foot sample as a first feature, and the first feature being a 64x1 audio amplitude vector;

[0088] Step S2.3: In a single two-foot sample, a handcrafted beatmap feature is extracted, and the handcrafted beatmap feature is input into a beatmap function to obtain a 64x70 feature matrix for representing the second feature;

[0089] The handcrafted beatmap feature includes a root mean square feature, a zero-crossing rate feature, a spectral center feature, and a spectral flatness feature.

[0090] Step S2.4: In a single two-foot sample, a mel-frequency spectrogram feature is extracted, and a 64x70 third feature is obtained by calculating a mel-frequency cepstral coefficient.

[0091] Step S2.5: Steps S2.1-S2.4 are repeated until all features of the single two-foot samples are extracted, and a two-foot step event sample library is established, and all extracted features are placed in the two-foot step event sample library.

[0092] Specifically, the step S3 includes:

[0093] Step S3.1: The features of the two-foot step event sample library are input into the three branch structures of the CRNN deep neural network for further feature extraction; after passing through the branch structure of the CRNN deep neural network, the first feature, the second feature, and the third feature are filtered, extracted, normalized, and pooled, so that the extracted features are more prominent and easier to identify.

[0094] The first branch structure uses a filter block as the first layer, each block is composed of a convolution layer, a batch normalization layer, and a max pooling layer for downsampling; the second branch structure is composed of two layers of GRU units for bidirectional calculation; the third branch structure is composed of four sub-blocks, each block is composed of a two-dimensional convolution layer, a batch normalization layer, and a max pooling layer.

[0095] Step S3.2: The extracted features are standardized and fused to obtain a fused feature.

[0096] Specifically, the end of each of the three branches adopts an adaptive average pooling layer to obtain output features of the same length through fusion, and the output features are combined in an element-wise summation manner to obtain the fused feature.

[0097] Specifically, the step S4 includes:

[0098] Step S4.1: The fused feature is input into a classification network for identity labeling.

[0099] Step S4.2: An identification model is established, and the labeled fused feature is input into the identification model for model training.

[0100] Embodiment 1

[0101] The present application provides a method for identifying the identity of footstep sound based on deep neural network, the main process is as shown in Figure 2 The main framework is as shown in Figure 3 The main process includes seven steps S1-S7:

[0102] S1: Use the microphone to record the data set in a less noisy environment;

[0103] S2: Divide the footstep events of the data set by finding local maximum value and setting minimum interval;

[0104] S3: Extract the features of the database audio with double footstep samples in three branches of waveform, beat figure and mel spectrum, and establish a double footstep event sample library;

[0105] S4: Input the three branch features extracted into CNN, RNN and CNN network respectively for further feature extraction, and then fuse them after extraction;

[0106] S5: Input the fused features into the classification network to obtain sample labels;

[0107] S6: Use the labeled double footstep event sample library to train the whole model;

[0108] S7: Collect footstep sound frequency data and extract fused features through S1-S5, and input them into the model for identity recognition.

[0109] The step S1 includes that a natural person with normal behavior ability walks naturally around a microphone with a center distance of about 80 cm in a room with less noise according to a circular trajectory, changes clothes and shoe type, and records 4 audio with a time length of about 60s.

[0110] The step S2 includes finding local maximum value of footstep sound frequency, using find_peaks function of SciPy library in Python to traverse data points and compare the size relationship of each data point and its adjacent points to determine which points are local maximum value. Set the distance threshold (minimum horizontal distance between adjacent peaks) to 400ms, take each peak value to offset 100ms as interval point, and divide all audio into footstep events;

[0111] The step S3 includes extracting waveform features, manual beatmap features and mel-spectrogram features respectively by taking the divided audio as a single sample with two footstep events.

[0112] The beatmap features can reflect the periodic structure in the audio signal, and have unique advantages when processing audio containing rhythmic information (such as the footstep sound concerned in the present application). In the prior art, since the beatmap contains more and more complex frequency feature information, and researchers pay more attention to the local features of the footstep sound, such as the cepstrum coefficient in the frequency domain, the research and application of the beatmap features are relatively less. The present application extracts the beatmap features in a specific range (the average range of the beat corresponding to the normal human footstep sound) and combines them with other features, effectively extracts the beatmap features from the audio signal without losing useful information, improves the expression ability of the audio features, and can more comprehensively describe the characteristics of the audio signal.

[0113] The accuracy of audio recognition is improved.

[0114]

[0115] wherein N is the number of samples; x[n] represents the amplitude of the signal at the nth sampling point;

[0116] Formula 1: RMS calculation formula:

[0117]

[0118] wherein sing(x) is a sign function.

[0119] Formula 2: ZCR calculation formula:

[0120]

[0121] wherein K is the number of frequency components, k is the frequency index, and magnitude[k] is the amplitude of the kth frequency component;

[0122] Formula 3: Spectral center calculation formula:

[0123]

[0124] where K is the number of frequency components, k is the frequency index, and magnitude[k] is the magnitude of the kth frequency component.

[0125] Formula 4: Spectral flatness calculation formula

[0126] Mel-spectrogram Mel-spectrogram is used to simulate the characteristics of human hearing in frequency, and librosa.feature.mfcc is used to calculate Mel-Frequency Cepstral Coefficients (MFCCs), to obtain a third feature with a shape of 64x70, which maintains the same matrix shape to facilitate subsequent network processing.

[0127] The step S4 includes inputting the obtained waveform features into a CNN network, and using a Sinc filter block as the first layer. In addition, each block is composed of a convolutional layer, a batch normalization layer, and a max-pooling layer for down-sampling; the hand-crafted features are input into an RNN network composed of two layers of GRU units, designed for bidirectional calculation; and the Mel-spectrogram features are input into a CNN network containing four sub-blocks, each composed of a two-dimensional convolutional layer, a batch normalization layer, and a max-pooling layer. An adaptive average pooling layer is used at the end of all three branches to obtain the same length of output features for subsequent fusion. Finally, the three features are combined in an element-wise summation manner.

[0128] The step S5 includes passing the fused features to a classification network composed of dense layers. Leaky ReLU is used as the activation function throughout the framework.

[0129] The step S6 includes inputting the double-footstep event sample library into the model for training. For data augmentation, the time shift randomly moves the waveform, hand-crafted audio features, and spectrogram along the time axis by 0 to 50%. We train the neural network from scratch with a batch size of 32 and use the SGD optimizer and a one-cycle learning rate scheduler. Cross-entropy loss is used as our loss function, and the three-cycle accuracy of the validation set no longer improves as the training stop flag.

[0130] The step S7 includes pre-processing the to-be-tested footstep audio, extracting features, inputting them into the model, and finally taking the most frequent identity of the events contained in the footstep audio as the recognition result.

[0131] The present application relates to a method and system for identifying the identity of footstep sound based on deep neural network. The method uses a microphone to collect data, filters the collected footstep sound, labels the data set, extracts the original waveform features, hand-crafted features and Mel spectrum features of the data set, and then puts them into a deep neural network (AFPI-ATM) for feature fusion and training. In particular, the present application first proposes to use beat map as a dimension of hand-crafted features. The training stage includes: footstep sound frequency collection, data preprocessing, extraction of specific features, and input into a deep neural network (CRNN, Convolutional and Recurrent Network). The identification stage includes: preprocessing of the footstep sound frequency to be tested, extracting its three main features, inputting them into the model, and finally taking the most frequent identity of the events contained in the footstep sound frequency as the identification result. The preprocessing part divides the footstep events of the footstep sound frequency; the audio feature extraction part first extracts the feature data of the three branches from the audio, and then inputs the network for processing and fusion into multi-modal features; the identity recognition part inputs the multi-modal features into AFPI-ATM, further extracts the footstep sound frequency features, and detects and identifies based on the existing library samples. In particular, the present application divides the footstep sound frequency according to single footstep event by finding local maximum value and setting minimum interval, extracts beat map as global feature of footstep sound frequency, and combines two footstep sound events into double footstep time as sample for identification. For the existing identities in the library, only two footstep sound times (about 1.4s) of audio can correctly distinguish them.

[0132] The present application also provides a system for identifying the identity of footstep sound based on deep neural network, which can be realized by executing the process steps of the method for identifying the identity of footstep sound based on deep neural network, i.e. the method for identifying the identity of footstep sound based on deep neural network can be understood by those skilled in the art as the preferred embodiment of the system for identifying the identity of footstep sound based on deep neural network.

[0133] According to the system for identifying the identity of footstep sound based on the deep neural network, the footstep sound frequency information is collected to form a database, and the data of the database is divided into footstep events; the footstep events are further divided, the waveform, the manual beat figure and the mel spectrum features in the footstep events are extracted, and a double footstep event sample library is established; the features of the double footstep event sample library are input into the CRNN deep neural network for further extraction, and the extracted features are standardized and fused to obtain the fused features; the fused features are used to train an identification model; and the footstep sound frequency information to be tested is input into the identification model, and the identity is identified by matching the information in the identification model.

[0134] Specifically, the module M1 comprises: a module M1.1, which collects the footstep sound frequency information within a preset range of a microphone and establishes a database; a module M1.2, which searches for the local maximum value of the footstep sound frequency every first time interval; and a module M1.3, which divides the footstep events; the local maximum value of the audio is taken as the starting point, and the starting time direction and the ending time direction are offset by a second preset time interval, respectively, and the two end points after the offset are taken as the starting point and the ending point of the footstep sound frequency time, and the audio information from the starting point to the ending point is a footstep event; wherein the second time interval is smaller than the first time interval, and the first time interval is the minimum horizontal distance between adjacent peak values; the module M1.2 to the module M1.3 are repeatedly triggered until all the audio information in the database is divided into footstep events, and the module M2 is entered.

[0135] Specifically, the module M2 comprises: a module M2.1, which takes the adjacent two footstep events as a single double foot sample according to the divided audio; a module M2.2, which extracts the waveform feature in the single double foot sample; specifically, the audio waveform in the single double foot sample is extracted as the first feature, and the first feature is a 64x1 audio amplitude vector; a module M2.3, which extracts the manual beat figure feature in the single double foot sample, and inputs the manual beat figure feature into a beat figure function to obtain a 64x70 feature matrix for representing the second feature; the manual beat figure feature comprises a root mean square feature, a zero-crossing rate feature, a spectral center feature and a spectral flatness feature; a module M2.4, which extracts the mel spectrum figure feature in the single double foot sample, and obtains a 64x70 third feature by calculating the mel frequency cepstrum coefficient; and a module M2.5, which repeatedly triggers the module M2.1 to the module M2.4 until the features of all the single double foot samples are extracted, and establishes a double footstep event sample library, and puts all the extracted features into the double footstep event sample library.

[0136] Specifically, the module M3 includes: module M3.1: inputting the features of the double-footstep event sample library into three branch structures of the CRNN deep neural network respectively to further extract features; wherein the first branch structure adopts a filter block as the first layer, each block is composed of a convolution layer, a batch normalization layer and a maximum pooling layer, and is used for down-sampling; the second branch structure is composed of two layers of GRU units, and is bidirectional calculation; the third branch structure is composed of four sub-blocks, each block is composed of a two-dimensional convolution layer, a batch normalization layer and a maximum pooling layer; module M3.2: standardizing and fusing the extracted features to obtain fused features; specifically, the end of the three branches adopts an adaptive average pooling layer, and the same length of output features is obtained through fusion, and the output features are combined in the form of element-wise summation to obtain the fused features.

[0137] Specifically, the module M4 includes: module M4.1: inputting the fused features into a classification network for identity labeling; module M4.2: establishing an identification model, and inputting the labeled fused features into the identification model for model training.

[0138] Those skilled in the art know that, in addition to implementing the system provided by the present application and each device, module and unit thereof in the form of pure computer readable program code, the system provided by the present application and each device, module and unit thereof can also be implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps to achieve the same functions. Therefore, the system provided by the present application and each device, module and unit thereof can be considered as a hardware component, and the devices, modules and units included therein for achieving various functions can also be considered as structures within the hardware component; the devices, modules and units for achieving various functions can also be considered as both software modules for implementing methods and structures within hardware components.

[0139] The specific embodiments of the present application are described above. It should be understood that the present application is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essential content of the present application. In the case of no conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for identifying a gait sound based on a deep neural network, characterized by, The application relates to a method for identifying the identity of a person based on footstep sound frequency information. The method comprises the following steps: Step S1: collecting footstep sound frequency information to form a database, and dividing the data in the database into footstep events; Step S2: further dividing the footstep events, extracting waveform, manual beat figure and mel-frequency spectrum features in the footstep events, and establishing a double-footstep event sample library; Step S3: inputting the features of the double-footstep event sample library into a CRNN deep neural network to further extract the features, and standardizing and fusing the extracted features to obtain fused features; Step S4: training an identification model by using the fused features; Step S5: inputting the to-be-detected footstep sound frequency information into the identification model, and identifying the identity by matching the information in the identification model; The step S1 comprises the following steps: Step S1.1: collecting footstep sound frequency information within a preset range of a microphone, and establishing a database; Step S1.2: searching for local maximum values of the footstep sound frequency every first time interval; Step S1.3: dividing the footstep events; taking the time at which the local maximum value of the audio is located as a starting point, offsetting the starting time direction and the ending time direction by a second preset time interval respectively, and taking the two end points after the offsetting as the starting point and the ending point of the footstep sound frequency time, the audio information from the starting point to the ending point being a footstep event; wherein the second time interval is smaller than the first time interval, and the first time interval is the minimum horizontal distance between adjacent peak values; Steps S1.2-S1.3 are repeated until all the audio information in the database is divided into footstep events, and step S2 is entered; The step S2 comprises the following steps: Step S2.1: taking two adjacent footstep events as a single double-foot sample according to the divided audio; Step S2.2: extracting waveform features in the single double-foot sample; specifically, extracting an audio waveform in the single double-foot sample as a first feature, and the first feature being a 64*1 audio amplitude vector; Step S2.3: extracting a manual beat figure feature in the single double-foot sample, inputting the manual beat figure feature into a beat figure function, and obtaining a 64*70 feature matrix to represent a second feature; The manual beat figure feature comprises a root-mean-square feature, a zero-crossing rate feature, a spectral center feature and a spectral flatness feature; Step S2.4: extracting a mel-frequency spectrum feature in the single double-foot sample, calculating a mel-frequency cepstral coefficient, and obtaining a 64*70 third feature; Step S2.5: repeating steps S2.1-S2.4 until the features of all the single double-foot samples are extracted, and establishing a double-footstep event sample library, and putting all the extracted features into the double-footstep event sample library; The step S3 comprises the following steps: Step S3.1: inputting the features of the double-footstep event sample library into three branch structures of a CRNN deep neural network to further extract the features; The first branch structure adopts a filter block as a first layer, each block is composed of a convolutional layer, a batch normalization layer and a maximum pooling layer, and is used for down-sampling; the second branch structure is composed of two GRU units, and is used for bidirectional calculation; and the third branch structure is composed of four sub-blocks, each block is composed of a two-dimensional convolutional layer, a batch normalization layer and a maximum pooling layer; Step S3.2: the extracted features are standardized and fused to obtain fused features; Specifically, the end of each of the three branches adopts an adaptive average pooling layer, and the same length of output features is obtained through fusion, and the output features are combined in an element-by-element summation manner to obtain fused features. 2.The method of claim 1, wherein, The step S4 includes: Step S4.1: inputting the fused features into a classification network for identity labeling; Step S4.2: establishing an identification model, and inputting the labeled fused features into the identification model for model training.

3. A system for identifying a footstep sound based on a deep neural network, characterized by, It includes: Module M1: collecting footstep audio information to form a database, and dividing the data in the database into footstep events; Module M2: further dividing the footstep events, extracting waveform, hand-timed beat figure and mel-frequency spectrum features in the footstep events, and establishing a double-footstep event sample library; Module M3: inputting the features of the double-footstep event sample library into a CRNN deep neural network for further extraction, and standardizing and fusing the extracted features to obtain fused features; Module M4: training an identification model by using the fused features; Module M5: inputting the to-be-tested footstep audio information into the identification model, and performing identity recognition by matching the information in the identification model; The module M1 includes: Module M1.1: collecting footstep audio information within a preset range of a microphone, and establishing a database; Module M1.2: searching for a local maximum value of the footstep audio every first time interval; Module M1.3: dividing footstep events; taking the time at which the local maximum value of the audio is located as a starting point, offsetting the starting time direction and the ending time direction by a second preset time interval respectively, and taking the two offset end points as the starting point and the ending point of the footstep audio time, the audio information from the starting point to the ending point is a footstep event; The second time interval is less than the first time interval, and the first time interval is the minimum horizontal distance between adjacent peak values; Repeat the module M1.2 to the module M1.3 until all audio information in the database is divided into footstep events, and then enter the module M2; The module M2 includes: Module M2.1: taking two adjacent footstep events as a single double-foot sample according to the divided audio; Module M2.2: extracting waveform features in the single double-foot sample; specifically, extracting an audio waveform in the single double-foot sample as a first feature, and the first feature is a 64x1 audio amplitude vector; Module M2.3: extracting a hand-timed beat figure feature in the single double-foot sample, inputting the hand-timed beat figure feature into a beat figure function, and obtaining a 64x70 feature matrix to represent a second feature; The hand-timed beat figure feature includes a root mean square feature, a zero-crossing rate feature, a spectral center feature and a spectral flatness feature; Module M2.4: In a single two-foot sample, extract the mel-spectrogram feature, and obtain a third feature of 64*70 by calculating the mel-frequency cepstral coefficient; Module M2.5: Repeat the triggering of modules M2.1-M2.4 until all the features of the single two-foot samples are extracted, and a two-foot step event sample library is established, and all the extracted features are put into the two-foot step event sample library; The module M3 comprises: Module M3.1: Input the features of the two-foot step event sample library into three branch structures of the CRNN deep neural network respectively to further extract the features; Wherein, the first branch structure adopts a filter block as the first layer, each block is composed of a convolution layer, a batch normalization layer and a maximum pooling layer, and is used for down-sampling; the second branch structure is composed of two layers of GRU units, and is bidirectional calculation; the third branch structure is composed of four sub-blocks, each block is composed of a two-dimensional convolution layer, a batch normalization layer and a maximum pooling layer; Module M3.2: Standardize and fuse the extracted features to obtain the fused features; Specifically, the end of each of the three branches adopts an adaptive average pooling layer, and the same length of output features is obtained by fusion, and the output features are combined in an element-by-element summation manner to obtain the fused features. 4.The system for identifying a footstep sound based on a deep neural network according to claim 3, wherein, The module M4 comprises: Module M4.1: Input the fused features into a classification network for identity labeling; Module M4.2: Establish an identification model, and input the labeled fused features into the identification model for model training.

Images