A multi-scale deep learning recognition method for sequence data

Abstract

The present invention discloses a multi-scale deep learning recognition method for sequence data, which collects and annotates sequence data to form an original sequence training data set, trains a modified deep neural network model, randomly intercepts data samples of a duration of t from the training samples of the original sequence training data set, and forward propagates the data into the deep neural network model. At the same time, the data of a duration of t is expanded or reduced and resampled to simulate time multi-scale, and the sampled data is forward propagated into the deep neural network model. The classification loss and feature loss of the two networks are combined to back-propagate the training neural network model. During the training process, the present invention simulates multi-scale phenomena by randomly resampling the training data within a certain range, thereby achieving higher recognition accuracy. The parameters of the two branch networks are shared during training, which effectively improves the training efficiency; only one branch network needs to be used in the recognition process, which is highly efficient.

Description

Technical Field

[0001] The present application belongs to the field of pattern recognition technology, and in particular relates to a multi-scale deep learning recognition method for sequence data. Background Art

[0002] In pattern recognition tasks for sequential data (such as audio and video), due to the different speaking and movement speeds of the target, the same content spans different time spans, that is, the multi-scale time problem, which brings great difficulty to recognition.

[0003] Currently, a common approach is to resample, stretch, or scale the data sequence to obtain multi-scale data. Then, a multi-branch network is used to process the data segments of different scales separately, and features of different scales are spliced ​​together for recognition, thereby reducing the impact of time scale on the recognition algorithm. This approach alleviates the impact of scale on the recognition algorithm to a certain extent, but this method of enumerating time scales can cause an overall time scale offset. Furthermore, the linear increase in the number of parameters in the multi-branch structure increases the difficulty of training and reduces recognition efficiency. Summary of the Invention

[0004] The purpose of this application is to provide a multi-scale deep learning recognition method for sequence data to solve the problems of low accuracy and efficiency in multi-scale recognition of sequence data. During the training process, a dual-branch network is used to integrate multi-scale thinking into the network parameters. The parameters of the two branch networks are shared, and one branch network is used in the recognition process to achieve accurate recognition of sequence data.

[0005] In order to achieve the above objectives, the technical solutions of this application are as follows:

[0006] A multi-scale deep learning recognition method for sequence data, comprising:

[0007] Collect and annotate sequence data to form the original sequence training data set;

[0008] Randomly intercept continuous data of duration t from the training samples of the original sequence training data set for sampling, forward propagate and input into the modified deep neural network model, the deep neural network model extracts a first feature, inputs the first feature into the recognizer, and calculates the first cross entropy loss between the recognizer output and the true label;

[0009] Randomly expand or reduce the data of time length t, and randomly resample it. The resampled data is forward propagated and input into the modified deep neural network model. The deep neural network model extracts the second feature, and the second feature is input into the recognizer. The second cross entropy loss between the recognizer output and the true label is calculated;

[0010] Calculating a feature difference between the first feature and the second feature, and calculating a weighted sum loss of the first cross entropy loss, the second cross entropy loss, and the feature difference, performing back propagation according to the weighted sum loss, and updating network parameters of the deep neural network model;

[0011] After the deep neural network model is trained, the trained deep neural network model is used for sequence data recognition.

[0012] Optionally, the sequence data is speech data of different people, and the annotation includes one of six emotions: anger, happiness, fear, sadness, surprise and neutral.

[0013] Optionally, the sequence data is video data of different actions of different people, and the annotation includes one of twenty-eight behaviors, including smiling, laughing, chewing, talking, smoking, eating, drinking, cartwheels, clapping, crawling, jumping, falling, flipping, handstand, jumping, pulling, pushing, running, sitting, somersaults, standing up, turning around, combing hair, grabbing, playing ball, riding a bicycle, hugging, and shaking hands.

[0014] Furthermore, the random resampling includes:

[0015] For each training sample data, take a random number r to obtain training sample data with a length that is r times the original length.

[0016] Furthermore, the feature difference between the first feature and the second feature is calculated using the following formula:

[0017]

[0018] Among them, L diff represents the feature difference between the first feature F1 and the second feature F2, and ||·||2 is the two-norm.

[0019] Optionally, the deep neural network model uses one-dimensional convolution to replace the two-dimensional convolution in the VGG16 network, uses an adaptive pooling operation to replace the last maximum pooling operation in the VGG16 network, and uses the last softmax layer of the VGG16 network as the recognizer.

[0020] Optionally, the deep neural network model adopts 3D Resnet, an adaptive pooling operation is used to replace the maximum pooling operation after the 5th res module of the 3DResnet, and the last softmax layer of the 3D Resnet is used as the recognizer.

[0021] Compared with the existing technology, the beneficial effects of the present application are: during the training process, the training data is randomly resampled within a certain range to simulate multi-scale phenomena, which will not cause the overall offset problem of the time scale and has higher recognition accuracy; the parameters of the two branch networks are shared, so the number of network parameters that need to be trained does not increase, which effectively improves the training efficiency; only one branch network is used in the recognition process, which is highly efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flowchart of the multi-scale deep learning recognition method for sequence data used in this application;

[0023] Figure 2 This is a block diagram of the neural network structure used in the training method of this application;

[0024] Figure 3 This is a block diagram of the neural network structure used in the post-training recognition process of this application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0026] In one embodiment, Figure 1 As shown, a multi-scale deep learning recognition method for sequence data is provided, comprising:

[0027] Step S1: Collect and annotate sequence data to form an original sequence training data set.

[0028] This application collects voice data of different people X i Each data is marked with one of the six emotions: anger, happiness, fear, sadness, surprise and neutral, and recorded as y i (y i ∈{0, 1, 2, 3, 4, 5}), a total of 10,000 sentences with different pronunciations were collected to form the training data set X = {X i ,y i} i＝1，2，...，N , where N=10000.

[0029] Step S2: Randomly intercept continuous data of duration t from the training samples of the original sequence training data set, forward propagate and input it into the modified deep neural network model, extract the first feature through the deep neural network model, input the first feature into the recognizer, and calculate the first cross entropy loss between the recognizer output and the true label.

[0030] In one embodiment, the deep neural network model constructed in the present application is an improvement based on VGG16. The deep neural network model uses one-dimensional convolution to replace the two-dimensional convolution in the VGG16 network, uses adaptive pooling operation to replace the last maximum pooling operation in the VGG16 network, and uses the last softmax layer of the VGG16 network as the recognizer.

[0031] Specifically, all 13 two-dimensional convolution operations in the VGG16 network structure were changed to one-dimensional convolution operations to accommodate the dimensionality requirements of the sound data. The kernel size of the convolution operation was set to 7, the stride to 2, and the padding to 3. The fifth maxpooling operation was changed to an adaptive pooling operation to ensure that input data of different lengths have the same size after the adaptive pooling operation. The last softmax layer of VGG16 is the recognizer C, and its output dimension is 6 (for a total of 6 emotions). The training parameters of the neural network were set: initial learning rate 0.01, training method Adam, step = 10000, batch_size = 100.

[0032] like Figure 2 As shown, the sample X in the original sequence training dataset X i Randomly intercept t seconds of continuous sound data (if X i If the number of features is less than t seconds, all features will be captured. The forward propagation enters the deep neural network model and the first feature F1 is obtained after the adaptive pooling operation P. The first feature F1 is passed to the recognizer C. The output of the recognizer C and the label y i The cross entropy loss is recorded as the recognition loss L1.

[0033] In a specific embodiment, the value of t is 2 seconds. The value of t is determined based on actual experimental results and is generally in the range of 0.5 to 5 seconds.

[0034] Step S3: Randomly expand or reduce the data of time length t, and randomly resample the data. The sampled data is forward propagated and input into the modified deep neural network model. The second feature is extracted by the deep neural network model, and the second feature is input into the recognizer. The second cross entropy loss between the recognizer output and the true label is calculated.

[0035] This step performs random resampling of the data for a duration of t during iterative training, including:

[0036] Each training sample data is randomly expanded or reduced, and randomly resampled, and a random number r is taken to obtain training sample data with a length that is r times the original length.

[0037] For example, for each training sample data, random numbers a and r are taken in the range of [0.5, 2], and random number b is taken in the range of [-0.5, 0.5]. Assume that the t-second data in step S2 is in the sample X i The time interval in is [T, T+t]. According to the time axis translation and expansion / contraction, we get a*t seconds of data in the time interval [T+(0.5+b-0.5a)t, T+(0.5+b+0.5a)t]. We perform random resampling to get training sample data X′ with a length r times the original length. i , to enrich the diversity of training data. Forward propagation enters the deep neural network model, and after the adaptive pooling operation P, the feature F2 is obtained. F2 is passed to the recognizer C, and according to the output of the recognizer C and the input data X i Label y i The difference between is used to calculate the recognition loss L2.

[0038] It should be noted that if Figure 2 As shown, step S3 and step S4 are trained separately through two branches respectively, and the parameters of the two branch networks are shared, so the number of network parameters that need to be trained does not increase, which effectively improves the training efficiency.

[0039] Step S4: Calculate the feature difference between the first feature and the second feature, and calculate the weighted sum loss of the first cross entropy loss, the second cross entropy loss, and the feature difference, perform back propagation according to the weighted sum loss, and update the network parameters of the deep neural network model.

[0040] This step calculates the difference between features F1 and F2 And calculate the total loss L = αL1 + βL2 + γL diff , and perform back propagation to update the parameters of the deep neural network model and the recognizer C.

[0041] In a specific embodiment, the value of α is 0.4, the value of β is 0.4, and the value of γ is 0.2. The values ​​of these weights can be obtained through experiments.

[0042] This application repeats the above training steps for a total of 10,000 iterations. After the training is completed, the parameters of the deep neural network model and recognizer C corresponding to the lowest recognition loss L recorded are used as the final parameters of the model to complete the model training.

[0043] After the deep neural network model training is completed, the trained deep neural network model can be used for emotion recognition, such as Figure 3Specifically, the following steps are performed: voice data X is collected, the middle 2 seconds of data is intercepted (if less than 2 seconds, the entire data is intercepted), forward propagated into the deep neural network model, and after adaptive pooling operation P, feature F1 is obtained. F1 is then passed to the recognizer C. The category corresponding to the maximum value among the six output components of recognizer C is the recognized emotion type.

[0044] In another embodiment, the sequence data of the present application can also be video image sequence data. For example, video data X of different people and different actions are collected. i Each data item is labeled with one of the 28 behaviors: smiling, laughing, chewing, talking, smoking, eating, drinking, cartwheel, clapping, climbing, jumping, falling, flipping, handstand, jumping, pulling, pushing, running, sitting, somersault, standing up, turning around, combing hair, grabbing, playing ball, riding a bicycle, hugging, shaking hands, and recorded as y. i (y i ∈{0, 1, 2, ..., 27}), a total of 10,000 videos were collected to form the dataset X = {X i ,y i} i＝1，2，...，N , where N=10000.

[0045] Select 3D ResNet as the network architecture for the deep neural network model. Replace the maxpooling operation after the fifth Res module with an adaptive pooling operation to accommodate input data of varying lengths. The final softmax layer of the 3D ResNet serves as the discriminator C, and its output dimension is modified to 28 emotion categories. Set the following neural network training parameters: initial learning rate 0.01, training method Adam, step size 2000, batch size 12.

[0046] Sample X in the original sequence data X i Randomly intercept 2 seconds of video (if X i If the video is less than 2 seconds, the entire video is captured). 32 frames of images are uniformly sampled from the video data to form an image sequence. Each frame is scaled to 224x224. The image sequence is forward propagated into the deep neural network model. After the adaptive pooling operation P, the feature F1 is obtained. F1 is passed to the recognizer C. The output of the recognizer C and the label y i The cross entropy loss is recorded as the recognition loss L1.

[0047] When processing each video in each iteration, for each training sample data, random numbers a and r are taken in the range of [0.5, 2], and random number b is taken in the range of [-0.4, 0.4]. Assuming that the 2-second data in step S2 is in sample X iThe time interval in the image is [T, T+2]. According to the time axis translation and expansion / contraction, 2a seconds of data are obtained for the time interval [T+2(0.5+b-0.5a), T+2(0.5+b+0.5a)]. Random resampling is performed to obtain 32*r frames (32*r is rounded to an integer) of images to form an image sequence. Each frame is scaled to 224x224 size and forward propagated into the deep neural network model. After the adaptive pooling operation P, the feature F2 is obtained. F2 is passed to the recognizer C. According to the output of the recognizer C and the input data X i Label y i The difference between is used to calculate the recognition loss L2.

[0048] Calculate the difference between features F1 and F2 where ||·||2 is the two-norm.

[0049] Calculate the total loss L = 0.42L1 + 0.42L2 + 0.18L diff , and backpropagation is performed to update the parameters of the deep neural network model and the recognizer. Repeat the above training steps for a total of 50,000 iterations; the parameters of the deep neural network and recognizer corresponding to the lowest recognition loss L recorded after training are used as the final parameters of the model.

[0050] After the deep neural network model training is completed, the trained deep neural network model can be used for action recognition. That is, video data X is collected, the middle 32 frames of data are intercepted (if there are less than 32 frames, all frames are intercepted), each frame image is scaled to 224x224 size, forward propagated into the deep neural network model, and after the adaptive pooling operation P, the feature F1 is obtained. F1 is passed to the recognizer C. The subscript of the maximum value among the 28 output components of the recognizer C is the action label.

[0051] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A multi-scale deep learning recognition method for sequence data, characterized in that: The multi-scale deep learning recognition method for sequence data includes: Collect and annotate sequence data to form the original sequence training data set; Randomly intercept continuous data of duration t from the training samples of the original sequence training data set for sampling, forward propagate and input into the modified deep neural network model, extract the first feature through the deep neural network model, input the first feature into the recognizer, and calculate the first cross entropy loss between the recognizer output and the true label; Randomly expand or reduce the data of time length t, and randomly resample it, forward propagate the resampled data and input it into the modified deep neural network model, extract the second feature through the deep neural network model, input the second feature into the recognizer, and calculate the second cross entropy loss between the recognizer output and the true label; Calculating a feature difference between the first feature and the second feature, and calculating a weighted sum loss of the first cross entropy loss, the second cross entropy loss, and the feature difference, performing back propagation according to the weighted sum loss, and updating network parameters of the deep neural network model; After the deep neural network model is trained, the trained deep neural network model is used for sequence data recognition; The deep neural network model uses one-dimensional convolution to replace the two-dimensional convolution in the VGG16 network, uses adaptive pooling to replace the last maximum pooling operation in the VGG16 network, and uses the last softmax layer of the VGG16 network as the recognizer. Alternatively, the deep neural network model adopts 3D Resnet, an adaptive pooling operation is used to replace the maximum pooling operation after the fifth res module of the 3D Resnet, and the last softmax layer of the 3D Resnet is used as the recognizer; The sequence data is speech data of different people, and the annotations include one of six emotions: anger, happiness, fear, sadness, surprise, and neutral; or the sequence data is video data of different actions of different people, and the annotations include one of twenty-eight actions: smiling, laughing, chewing, talking, smoking, eating, drinking, cartwheels, clapping, crawling, jumping, falling, flipping, handstand, jumping, pulling, pushing, running, sitting, somersaults, standing up, turning around, combing hair, grabbing, playing ball, riding a bicycle, hugging, and shaking hands.

2. The multi-scale deep learning recognition method for sequence data according to claim 1, characterized in that The random resampling comprises: For each training sample data, take a random number r to obtain training sample data with a length that is r times the original length.

3. The multi-scale deep learning recognition method for sequence data according to claim 1, characterized in that The feature difference between the first feature and the second feature is calculated using the following formula: ； in, Represents the first feature and the second feature The characteristic differences, is the two-norm.

Images