Network training method and device, and electronic equipment

Abstract

The embodiment of the invention provides a network training method, which is used for training gesture recognition and comprises the following steps: acquiring a training data set of dynamic gestures; designing a multilayer convolutional neural network training model; and training the training data set through the multi-layer convolutional neural network training model. According to the network training method provided by the embodiment of the invention, the training data set of the dynamic gesture is trained through the multi-layer convolutional neural network training model, so that the accuracy of the training result of the dynamic gesture is improved. The embodiment of the invention further provides a network training device and electronic equipment.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of deep learning technology, and specifically, to a network training method, device, and electronic device. Background Art

[0002] When users are watching videos, the viewing experience will be affected due to the limitations of external devices such as keyboards and remote controls. Therefore, dynamic gesture recognition is applied to video playback. When users are watching videos, dynamic gesture control of video playback not only adds fun, but also expands the user's freedom, freeing them from the constraints of external devices such as remote controls.

[0003] Convolutional neural networks can effectively learn the features brought by a single image. By processing and training the data set, the overall features of the image can be planned as a unit feature, which is convenient for the subsequent correspondence between the unit feature and the label. Single-layer convolutional neural networks are difficult to learn dynamic gesture images with time effects. Dynamic gesture images are a set of continuous frames, and the frames are interconnected. It is difficult to achieve the desired effect by only identifying dynamic gestures from a single frame of image. Summary of the invention

[0004] In response to the problems existing in the above-mentioned prior art, the embodiments of the present application provide a network training method, device, and electronic device, which trains a training data set of dynamic gestures through the multi-layer convolutional neural network training model to improve the accuracy of the training results of dynamic gestures.

[0005] In a first aspect, an embodiment of the present application provides a network training method for training gesture recognition, comprising the following steps:

[0006] Get a training dataset of dynamic gestures;

[0007] Design a multi-layer convolutional neural network training model; and

[0008] The training data set is trained by the multi-layer convolutional neural network training model.

[0009] Furthermore, the multi-layer convolutional neural network training model includes a 3D convolutional neural network layer, a maximum pooling layer and a recurrent convolution layer.

[0010] Furthermore, the training data set is trained by the multi-layer convolutional neural network training model, including:

[0011] Constructing multiple sets of training data sets, and training the multiple sets of training data sets by using the multi-layer convolutional neural network training model; and

[0012] The prediction results of the multiple sets of training data sets are compared to determine whether they belong to the same gesture.

[0013] Further, after the training data set is trained by the multi-layer convolutional neural network training model, the method further includes:

[0014] Add a finite state machine to make a connection between the input gesture and the current gesture to derive the expected gesture.

[0015] Furthermore, after obtaining the training data set of dynamic gestures, the method further includes:

[0016] The training data set is classified according to the category of the dynamic gesture.

[0017] Furthermore, the optimizer of the multi-layer convolutional neural network training model adopts stochastic gradient descent method.

[0018] Furthermore, the training of the training data set by the multi-layer convolutional neural network training model further includes:

[0019] Through multi-layer perception, the residual connection method is used to establish the connection between high-feature gestures and low-feature gestures.

[0020] In a second aspect, the embodiment of the present application further provides a network training device, including:

[0021] A data acquisition module, used to acquire a training data set of dynamic gestures;

[0022] A model design module for designing a multi-layer convolutional neural network training model; and

[0023] A data training module is used to train the training data set through the multi-layer convolutional neural network training model.

[0024] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to implement the network training method according to the first aspect when executing the program.

[0025] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, wherein the computer program is used to implement the network training method according to the first aspect above.

[0026] The embodiments of the present application bring the following beneficial effects:

[0027] In the network training method provided in the embodiment of the present application, a training data set of dynamic gestures is first obtained, and then a multi-layer convolutional neural network training model is designed, and the training data set is trained by the multi-layer convolutional neural network training model. The network training method provided in the embodiment of the present application trains the training data set of dynamic gestures by the multi-layer convolutional neural network training model to improve the accuracy of the training results of dynamic gestures. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0029] Figure 1 A flowchart of a network training method provided in an embodiment of the present application;

[0030] Figure 2 for Figure 1 A schematic diagram of the multi-layer information of the training model provided by the network training method;

[0031] Figure 3 A structural block diagram of a network training device provided in an embodiment of the present application;

[0032] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0033] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments described in the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0035] In the specification and claims of this application and the above-mentioned drawings, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "multiple" means two or more. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to the specific circumstances.

[0036] Figure 1 FIG. 1 is a flow chart of a network training method according to an embodiment of the present application. Figure 1 As shown, the network training method of the embodiment of the present application is used for training gesture recognition, and includes the following steps:

[0037] S101: Obtain a training data set of dynamic gestures;

[0038] A dynamic gesture image is a set of continuous frames, and the frames are interconnected. The embodiment of the present application can use the public dataset Jester as the dataset of this model. When the dataset is obtained, it is not already classified into each dynamic gesture class. The gesture category corresponding to each data is labeled by the downloaded hand_gestures_labels.csv. During training, this file is needed to match the labels to the data one by one.

[0039] When selecting a specific gesture for training, the data set needs to be classified. The gestures required by the embodiment of the present application are swiping left, swiping right, thumb up, thumb down, clicking fingers, shaking wrists, two fingers spread out, two fingers merged, and no gesture, a total of nine gestures. Determine the sequence number by checking the key value on hand_gestures_labels.csv, and then determine which folder it belongs to by its value. Finally, the nine gestures in the data set are separated to facilitate subsequent model training. In an embodiment of the present application, only 18 frames of gestures of each participant can be extracted, and the gestures of 700 participants can be extracted at the same time as training data for its network model.

[0040] S102: Design a multi-layer convolutional neural network training model; and

[0041] As described above, it is difficult for a single-layer convolutional neural network to learn dynamic gesture images with time effects. Therefore, the embodiment of the present application constructs a multi-layer convolutional neural network training model to train the training data set of dynamic gestures.

[0042] S103: Training the training data set using the multi-layer convolutional neural network training model.

[0043] After the multi-layer convolutional neural network training model is built, the dynamic gesture training data set needs to be input for training.

[0044] Therefore, in the network training method provided in the embodiment of the present application, a training data set of dynamic gestures is first obtained, and then a multi-layer convolutional neural network training model is designed, and the training data set is trained through the multi-layer convolutional neural network training model. The network training method provided in the embodiment of the present application trains the training data set of dynamic gestures through the multi-layer convolutional neural network training model to improve the accuracy of the training results of dynamic gestures.

[0045] Further, refer to Figure 2 In some embodiments of the present application, the multi-layer convolutional neural network training model includes a 3D convolutional neural network layer (Conv 3D), a maximum pooling layer (Max pool) and a recurrent convolution layer (ConvLSTM), so as to further improve the accuracy of the training results of dynamic gestures.

[0046] Specifically, 3D convolutional neural networks are more suitable for learning spatiotemporal features than 2D convolutional neural networks. Secondly, the network structure with a 3×3× convolution kernel size in each layer of the 3D convolutional neural network can achieve the best performance. Finally, the learned feature sequence is strengthened in the recurrent convolution layer ConvLSTM to strengthen the correlation between its feature maps. In this training model, the feature map of the 2D output is compared with the feature map output by this model, and the feature map of the training model shows the learning content between the connected frames. Therefore, this training model can learn spatiotemporal features better. The structure of the network will be described layer by layer below.

[0047] First, the first layer is a 3D convolutional neural network layer, which has 32 neurons, that is, it will output 32 feature maps, the size of the convolution kernel is (3, 3, 3), and the standard value of the data input data is (18, 50, 50, 3), which are the number of frames, the length of the image, the width of the image, and the three color channels of the image RGB. Although the time series is convolved, by setting its padding = `same`, the size of the input image and the length of the sequence are not changed during the convolution. After the convolution, the feature map is still 50×50 and the time series is still 18. It is concluded through the computer that a feature map calculation in this layer will have (3×3×3×3+1) kernel parameters with 32 at the same time, and a total of 2624 weight parameters.

[0048] The second layer is a maximum pooling layer, which performs maximum pooling of scale (1, 2, 2) on the rows, columns and time dimensions of the feature map. After the second maximum pooling layer, the tensor is calculated to be (18, 25, 25), where 18 is the time series of the feature map and 25×25 is the size of the removed feature map. The number of feature maps is still 32.

[0049] The parameters of the third and fourth layers are basically the same as those of the first and second layers, except that the combination of the two layers has 64 neurons. Finally, through calculation, the number of weights in the 3D convolutional neural network layer is 55360.

[0050] The combination between the fifth and sixth layers is similar to that between the previous layers, except that the number of feature maps between them is 128, and the number of weights calculated is 221312. The final tensor of this layer is (18, 6, 6, 128), where 18 is still the length of its time series, 6×6 is the size of its feature map, and 128 is the number of feature maps.

[0051] The seventh, eighth and ninth layers also use 3D convolutional neural network layers, but the difference from the previous layers is that these three layers do not use the maximum pooling layer to enhance their learning ability. The number of neurons in these three layers is 256, and the size of the convolution kernel is (3, 3, 3). The number of weights is calculated to be 884992, 1769728 and 1769728. At the same time, its output tensor is (18, 6, 6, 256).

[0052] The tenth layer performs maximum pooling on the feature map output by the previous layer. After passing through the pooling layer, the output tensor changes to (18, 3, 3, 256).

[0053] The 11th and 12th layers both use the recurrent convolutional layer ConvLSTM layer to process the time series. The connection between consecutive images is strengthened so that the model can learn the connection between consecutive frames and thus achieve the ability to recognize dynamic gestures. The weight parameters of this layer are all 4719616.

[0054] The thirteenth layer is the global max pool layer. The main purpose of this layer is to flatten the feature map of the previous volume to facilitate the mechanical connection of the fully connected layer to the classifier, which is the last layer to the output layer.

[0055] Finally, the fully connected layer inputs the output vector of the previous layer into a softmax classifier to classify the gestures. The output of this layer is the proportion of each gesture.

[0056] In the training model provided in the embodiment of the present application, for multi-classification problems, the loss function of the neural network model uses the classification cross entropy loss function, namely categorical_crossentropy, and the entire model multi-optimizer uses the stochastic gradient descent method (Stochastic Gradient Descent, SGD) in this paper. The initial learning rate of the stochastic gradient descent algorithm is set to 0.001. When it iterates ten times and the accuracy does not increase, the learning rate is adjusted to 0.00083333. In this model, the batch size (Batch Size) of the neural network reading the data set during training is selected as 32, that is, each time a five-dimensional tensor with a shape of 32×18×50×50×3 is read, and the corresponding label (label) in the 32 gesture frames is read, and the shape is a one-dimensional tensor of 32×1, and these data are passed to their own model in the form of "sample-label" for training the neural network model. It should be noted that when reading the label, the label needs to be mapped to a binary category matrix. When actually acquiring data, we directly read the data processed in the data preprocessing stage, that is, first select the first 18 frames from each folder in each dynamic gesture as the training data set, and adjust its size to 50×50 by reading the picture, that is, its tensor is (18, 50, 50, 3). During gradient descent, the number of samples contained in each Batch is set to 32, which means that the gradient descent algorithm will be performed once for every 32 samples during the training process, and the objective function will be optimized after updating. The number of training rounds (epochs) is set to 100, that is, the training will stop when the value of this round is reached. In order to prevent too many training data times and prevent overfitting, the ModelCheckpoint function is used to save the optimized model in the training process when training the model. During the training process, when the accuracy of the model sample trained at this time is higher than the accuracy obtained last time, this time is saved locally.

[0057] Furthermore, in some embodiments of the present application, the training data set is trained by the multi-layer convolutional neural network training model, including:

[0058] Constructing multiple sets of training data sets, and training the multiple sets of training data sets by using the multi-layer convolutional neural network training model; and

[0059] The prediction results of the multiple sets of training data sets are compared to determine whether they belong to the same gesture.

[0060] Specifically, since a single model may cause misjudgment of similar gestures, which may cause certain errors in video control, two models may be constructed using multiple data sets, such as two data sets, and then the prediction values ​​of the two models may be used to determine whether they are the same gesture. For example, the data set of the first model is trained on the first eighteen frames of each dynamic gesture, and the data set of the second model is trained on the middle eighteen frames of each dynamic gesture, thereby avoiding misjudgment of similar dynamic gestures.

[0061] Furthermore, in some embodiments of the present application, after the training data set is trained by the multi-layer convolutional neural network training model, the method further includes:

[0062] Add a finite state machine to make a connection between the input gesture and the current gesture to derive the expected gesture.

[0063] Specifically, after the training model is recognized, in the user identification process that needs to be applied, the corresponding video operation label can be set for each gesture. The fast forward and fast rewind of the video correspond to the gestures of clicking the finger and shaking the wrist, the previous episode and the next episode of the video correspond to sliding to the left and sliding to the right, the volume increase and decrease of the video correspond to thumb up and thumb down, and the full screen and zoom of the video correspond to two fingers spreading and two fingers merging. Because the control of the video is linked to the context, in order to ensure the connection between the contexts, the embodiment of the present application adds a finite state machine, that is, the gesture that needs to be input is ci, which is connected to the gesture s of the current state, and the gesture g is calculated, which indicates that the gesture of the next state depends on the context, that is, when in the video playback state, only the pause, rewind, forward and volume adjustment functions can be used, and it can no longer play.

[0064] Furthermore, in some embodiments of the present application, after acquiring the training data set of dynamic gestures, the method further includes:

[0065] The training data set is classified according to the category of the dynamic gesture.

[0066] As mentioned above, when selecting a specific gesture for training, the data set needs to be classified. The gestures required by the embodiment of the present application are swiping left, swiping right, thumb up, thumb down, clicking fingers, shaking wrists, two fingers spread out, two fingers merged, and no gesture, a total of nine gestures. Determine the sequence number by checking the key value on hand_gestures_labels.csv, and then determine which folder it belongs to by its value. Finally, separate the nine gestures in the data set to facilitate subsequent model training.

[0067] Furthermore, in some embodiments of the present application, the optimizer of the multi-layer convolutional neural network training model adopts the stochastic gradient descent method to further improve the accuracy of model training.

[0068] Furthermore, in some embodiments of the present application, the training of the training data set by the multi-layer convolutional neural network training model further includes:

[0069] Through multi-layer perception, the residual connection method is used to establish the connection between high-feature gestures and low-feature gestures.

[0070] Specifically, in the network training method provided in the embodiment of the present application, a multi-layer perception method can be used to enhance the relationship between frame features of dynamic gestures, which can effectively increase the recognition accuracy of the model. Through the multi-layer perception method, residual connections are used to ensure information exchange between high features and low features for processing on a 3D convolutional network.

[0071] Figure 3 2 is a structural block diagram of a network training device 200 provided in an embodiment of the present application. Figure 3 As shown, the network training device 200 of the embodiment of the present application includes: a data acquisition module 210, a model design module 220 and a data training module 230, wherein:

[0072] A data acquisition module 210 is used to acquire a training data set of dynamic gestures;

[0073] A model design module 220, for designing a multi-layer convolutional neural network training model; and

[0074] The data training module 230 is used to train the training data set through the multi-layer convolutional neural network training model.

[0075] In the network training device provided in the embodiment of the present application, a training data set of dynamic gestures is first obtained, and then a multi-layer convolutional neural network training model is designed, and the training data set is trained by the multi-layer convolutional neural network training model. The network training method provided in the embodiment of the present application trains the training data set of dynamic gestures by the multi-layer convolutional neural network training model to improve the accuracy of the training results of dynamic gestures.

[0076] It should be noted that the specific implementation of the network training device of the embodiment of the present application is similar to the specific implementation of the network training method of the embodiment of the present application. Please refer to the description of the method part for details, and no further details will be given here.

[0077] Figure 4 Schematic diagram of the structure of an electronic device 300 according to an embodiment of the present application.

[0078] like Figure 4 As shown, the electronic device 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from the storage part 302 to a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The CPU 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0079] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, etc.; an output section 307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed, so that a computer program read therefrom is installed into the storage section 308 as needed.

[0080] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 309, and / or installed from a removable medium 311. When the computer program is executed by a central processing unit (CPU) 301, the above-mentioned functions defined in the electronic device of the present application are executed.

[0081] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electronic device, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0082] In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction-executing electronic device, apparatus, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction-executing electronic device, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0083] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the processing receiving device, method and computer program product according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the aforementioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based electronic device that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0084] The units or modules involved in the embodiments described in this application may be implemented by software or hardware. The units or modules described may also be set in a processor, and the processor is used to implement the network training method when executing the program:

[0085] Get a training dataset of dynamic gestures;

[0086] Design a multi-layer convolutional neural network training model; and

[0087] The training data set is trained by the multi-layer convolutional neural network training model.

[0088] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently and not be assembled into the electronic device. The above computer-readable storage medium stores one or more programs, and when the above programs are used by one or more processors to execute the network training method described in the present application:

[0089] Get a training dataset of dynamic gestures;

[0090] Design a multi-layer convolutional neural network training model; and

[0091] The training data set is trained by the multi-layer convolutional neural network training model.

[0092] As another aspect, the present application further provides a computer program product, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer program product stores one or more programs, and when the above programs are used by one or more processors to execute the network training method described in the present application:

[0093] Get a training dataset of dynamic gestures;

[0094] Design a multi-layer convolutional neural network training model; and

[0095] The training data set is trained by the multi-layer convolutional neural network training model.

[0096] The above description is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. All equivalent structural changes made by using the contents of the present application specification and drawings under the application concept of the present application, or directly / indirectly used in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A network training method for training gesture recognition, characterized in that: The following steps are involved: Get a training dataset of dynamic gestures; Design a multi-layer convolutional neural network training model; and The training data set is trained by the multi-layer convolutional neural network training model.

2. The network training method according to claim 1, characterized in that: The multi-layer convolutional neural network training model includes a 3D convolutional neural network layer, a maximum pooling layer and a recurrent convolution layer.

3. The network training method according to claim 2, characterized in that: The training of the training data set by using the multi-layer convolutional neural network training model includes: Constructing multiple sets of training data sets, and training the multiple sets of training data sets by using the multi-layer convolutional neural network training model; and The prediction results of the multiple sets of training data sets are compared to determine whether they belong to the same gesture.

4. The network training method according to claim 1, characterized in that: After the training data set is trained by the multi-layer convolutional neural network training model, the method further includes: Add a finite state machine to make a connection between the input gesture and the current gesture to derive the expected gesture.

5. The network training method according to claim 1, characterized in that: After obtaining the training data set of dynamic gestures, the method further includes: The training data set is classified according to the category of the dynamic gesture.

6. The network training method according to claim 1, characterized in that: The optimizer of the multi-layer convolutional neural network training model adopts the stochastic gradient descent method.

7. The network training method according to claim 1, characterized in that: The training of the training data set by the multi-layer convolutional neural network training model further includes: Through multi-layer perception, the residual connection method is used to establish the connection between high-feature gestures and low-feature gestures.

8. A network training device, characterized in that: include: A data acquisition module, used to acquire a training data set of dynamic gestures; Model design module, used to design multi-layer convolutional neural network training models; and A data training module is used to train the training data set through the multi-layer convolutional neural network training model.

9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor is used to implement the network training method according to any one of claims 1 to 7 when executing the program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is used to implement the network training method according to any one of claims 1-7.

Images