Multi-channel action recognition method and device based on adaptive convolutional neural network

Abstract

The application discloses a multi-channel action recognition method and device based on an adaptive convolutional neural network, and the method comprises the following steps: acquiring time sequence data of an action collected by multiple sensors, pre-processing the time sequence data to obtain multiple sensor data; inputting the multiple sensor data into an action recognition model respectively, wherein one kind of sensor data corresponds to one channel of the action recognition model, output data of multiple channels is obtained, the action recognition model comprises an adaptive convolutional neural network, a GRU gate recurrent neural network and a multi-head self-attention mechanism layer which are connected in sequence; and the output data of the multiple channels is merged and passed through a classifier to obtain an action recognition result. The data of multiple source sensors is processed separately by using multiple channels, the features of different actions extracted have smaller granularity, the correlation between each time node in a complete time segment is focused on, the long-time forgetting problem can be solved, and the parallel processing mode improves the calculation efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of time series classification in deep learning, and in particular to a multi-channel action recognition method and device based on an adaptive convolutional neural network. BACKGROUND

[0002] In recent years, with the gradual popularization of AI, VR, AR technology and consumption upgrading, wearable devices (including smart watches, bracelets, 3D glasses, etc.) have gradually developed from single function to multi-function, while maintaining portability and practicality. Based on the rich sensors in wearable devices, we can decode human activities, which is very helpful for health monitoring, daily behavior analysis, smart home, abnormal behavior detection and other research scenarios in the medical field. Therefore, human activity recognition (HAR) through obtaining data from various sensors is attracting more and more attention.

[0003] HAR can be roughly divided into two types: one is based on video images, and the other is based on sensors. The video image-based system uses cameras or surveillance cameras to record a series of human action pictures, and identifies the types of actions. The sensor-based system uses passive radar signals or active sensor data to identify human activities. Thanks to the popularity of smart wearable devices and smartphones, their portability and practicality, and the embedding of various micro sensors and continuous recording of human-related information, sensor-based human activity recognition has gradually occupied a dominant position.

[0004] In recent years, some machine learning techniques (Machine Learning, ML) have been applied to HAR. Including Support Vector Machine (SVM), Random Forest (RF), Decision Tree (DT) and game theory-based feature selection method, etc. Deep learning (Deep Learning, DL) is a branch of machine learning, which has achieved great success in computer vision, natural language processing and other fields. Some deep learning-based human activity recognition methods such as Convolutional Neural Networks (CNN), Deep Belief Networks (DBN), Recurrent Neural Networks (RNN) and others overcome the shortcomings of manual feature extraction, can automatically detect features and almost do not need human intervention, and can well identify human activities.

[0005] Existing technologies also have some drawbacks, mainly the following:

[0006] (1) Machine learning-based technologies require a lot of effort in data preparation, data preprocessing and feature extraction, and require manual screening and processing of features.

[0007] (2) Traditional CNN models only extract features within a fixed small range, while truly meaningful data is usually a fragment rather than a single point in time.

[0008] (3) Traditional RNN models suffer from the problem of vanishing gradients and cannot handle long-term dependencies. This leads to the loss of a series of feature information and makes it impossible to focus on the correlation between various time points in a complete time period.

[0009] (4) Under the condition of multiple sensors, for some very similar actions, the previous models did not process these sensors separately. Instead, they mixed these data together and fed them into the model, resulting in the loss of some features that could have been used to distinguish similar actions. Summary of the Invention

[0010] In view of the technical problems existing in the background art mentioned above, the purpose of the embodiments of this application is to propose a multi-channel action recognition method and apparatus based on an adaptive convolutional neural network. This method uses multiple channels to process the sensor data from multiple sources separately, extracting features of different actions in a more granular manner, thereby solving the technical problems mentioned in the background art section above.

[0011] In a first aspect, embodiments of this application provide a multi-channel action recognition method based on an adaptive convolutional neural network, comprising the following steps:

[0012] S1, acquire time-series data of an action collected by multiple sensors, preprocess the time-series data to obtain data from multiple sensors;

[0013] S2, input multiple sensor data into the action recognition model respectively, where one type of sensor data corresponds to one channel of the action recognition model, and obtain the output data of multiple channels. The action recognition model includes an adaptive convolutional neural network, a GRU gated recurrent neural network and a multi-head self-attention mechanism layer connected in sequence.

[0014] S3 merges the output data from multiple channels and passes it through a classifier to obtain the action recognition result.

[0015] In some embodiments, the time series data in step S1 undergoes preprocessing, specifically including:

[0016] S11, a sliding window method is used to cut the time series data into fixed-size motion data; the size of each motion data is C×T; where C is the number of sensor channels and T is the time span of the motion;

[0017] S12 performs zero-padding, noise reduction, and standardization on the motion data, and splits it according to different types of sensors to obtain multiple sensor data.

[0018] In some embodiments, the adaptive convolutional neural network has multiple convolutional kernels, each with a different size.

[0019] In some embodiments, the adaptive convolutional neural network in the action recognition model of step S2 is specifically implemented as follows:

[0020] S21. The input data X = input∈R obtained by transposing the sensor data. T×C The input data is fed into an adaptive convolutional neural network, which includes a convolution operation Conv(·), a batch normalization (BN) operation, a dropout operation, and an activation function ReLU(·); the convolution operation contains a set of convolution kernels K = {k1, k2, k3…k}. m}, k i This represents a convolutional kernel with the value of length, where i ∈ {1, 2, 3…m}, m is the number of convolutional kernels, and the size of each convolutional kernel is k. i ×C, each convolutional kernel k i The corresponding convolution kernel output is P i :

[0021]

[0022] in, This is the weight matrix in the convolution operation. d o The dimension of the channel output;

[0023] S22, output P from all convolution kernels. i The output P is obtained by merging the results.

[0024] P = {P1, P2, P3…P} m};

[0025] The output P is passed through two fully connected layers and a softmax layer to calculate the importance score:

[0026] score = softmax(FFN(P) i ));

[0027]

[0028] wherein FFN is a two-layer fully connected layer, and is a weight matrix of the fully connected layer, and b1 and b2 are bias terms corresponding thereto;

[0029] S23, each output result P i is weighted and fused with the importance score score i corresponding thereto to obtain a merged output P score =∑ i score i ×P i , wherein P score ∈R T×C , i∈{1,2,3…m}, and the merged output P score is then residual connected with the input data X to obtain a first-layer output result P out =X+P score , P out ∈R T×C .

[0030] In some embodiments, the GRU gated recurrent neural network in the action recognition model of step S2 is embodied as follows: the first-layer output result P out ∈R T×C is split according to the dimension T to obtain P out ={M1,M2…M T}, each M t ∈R 1×C is input to the GRU gated recurrent neural network, and a second-layer output result H t is output, wherein t={1,2,3…T}.

[0031] In some embodiments, the multi-head self-attention mechanism layer in the action recognition model of step S2 is embodied as follows:

[0032] All the second-layer output results H t are merged into a matrix H={H1,H2,H3…H T}, a multi-head attention mechanism is implemented based on the matrix H, and a third-layer output result is output:

[0033] Atten channeln =[head1;head2;…head h ]W O ;

[0034] wherein each head is calculated as follows:

[0035]

[0036]

[0037]

[0038]

[0039] wherein, W O are weight matrices of Q, K, V and Attention respectively, j e {1, 2, 3…h}; d k is dimension, and n is the number of channels of the action recognition model.

[0040] In some embodiments, step S3 specifically comprises:

[0041] The output results of the third layer of each channel of the action recognition model are merged, and pass through a fully connected layer and a softmax layer to obtain an action recognition result:

[0042] output concat =[Atten channel 1 ;Atten channel 2 ;…Atten channeln ];

[0043] output=softmax(FFN(output concat ))。

[0044] In a second aspect, embodiments of the present application provide a multi-channel action recognition device based on an adaptive convolutional neural network, comprising:

[0045] A data acquisition module configured to acquire time series data of an action collected by multiple sensors, and preprocess the time series data to obtain multiple sensor data;

[0046] A neural network module configured to input the multiple sensor data into an action recognition model respectively, wherein one kind of sensor data corresponds to one channel of the action recognition model, and output data of multiple channels are obtained, the action recognition model comprising an adaptive convolutional neural network, a GRU gated recurrent neural network and a multi-head self-attention mechanism layer connected in sequence;

[0047] A classification module configured to merge the output data of the multiple channels and pass through a classifier to obtain an action recognition result.

[0048] In a third aspect, an embodiment of the present application provides an electronic device, comprising one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the implementation manners of the first aspect.

[0049] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, having stored thereon a computer program, when the computer program is executed by a processor, the method described in any of the implementation manners of the first aspect is implemented.

[0050] Compared with the prior art, the present application has the following beneficial effects:

[0051] (1) The present application constructs a multi-channel action recognition model for processing multi-source sensor data input. Each sensor is processed separately, and data features are extracted in a more fine-grained manner.

[0052] (2) The present application uses an adaptive convolutional neural network to extract features of different length segments on the time axis, so that the action recognition model no longer performs convolution operation only on a single time point, and then the features extracted by different convolution kernels are weighted and fused after the action recognition model learns to judge their importance.

[0053] (3) The present application uses a multi-head self-attention mechanism to process time series, and the action recognition model can pay attention to the correlation between each time node in the complete time segment, solving the long-time forgetting problem caused by only using a recurrent neural network, and the parallel processing mode improves the computing efficiency of the action recognition model. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0055] Figure 1 is an exemplary device architecture diagram to which an embodiment of the present application can be applied;

[0056] Figure 2 is a flowchart of the multi-channel action recognition method based on the adaptive convolutional neural network of the embodiment of the present application;

[0057] Figure 3 is a schematic diagram of the overall flow framework of the multi-channel action recognition method based on the adaptive convolutional neural network of the embodiment of the present application;

[0058] Figure 4 A structural diagram of an action recognition model of an embodiment of the adaptive convolutional neural network based multi-channel action recognition method of the present application;

[0059] Figure 5 A schematic diagram of an embodiment of the adaptive convolutional neural network based multi-channel action recognition device of the present application;

[0060] Figure 6 A structural diagram of a computer device suitable for implementing the electronic device of the embodiments of the present application. DETAILED DESCRIPTION

[0061] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort belong to the scope of protection of the present application.

[0062] Figure 1 An exemplary device architecture 100 of the adaptive convolutional neural network based multi-channel action recognition method or the adaptive convolutional neural network based multi-channel action recognition device of the embodiments of the present application is shown.

[0063] As shown in Figure 1 , the device architecture 100 can include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a communication link medium between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0064] A user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various applications can be installed on the terminal devices 101, 102, 103, such as data processing type applications, file processing type applications, etc.

[0065] The terminal devices 101, 102, 103 can be hardware or software. When the terminal devices 101, 102, 103 are hardware, they can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers and desktop computers, etc. When the terminal devices 101, 102, 103 are software, they can be installed in the above-mentioned listed electronic devices. They can be implemented as multiple software or software modules (such as software or software modules used to provide distributed services), or as a single software or software module. No specific limitation is made herein.

[0066] The server 105 can be a server providing various services, such as a background data processing server processing files or data uploaded by the terminal devices 101, 102, and 103. The background data processing server can process the obtained files or data to generate a processing result.

[0067] It should be noted that the multi-channel action recognition method based on an adaptive convolutional neural network provided in the embodiments of the present application can be executed by the server 105, or can be executed by the terminal devices 101, 102, and 103. Correspondingly, the multi-channel action recognition apparatus based on an adaptive convolutional neural network can be arranged in the server 105, or can be arranged in the terminal devices 101, 102, and 103.

[0068] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the above apparatus architecture is merely illustrative. Any number of terminal devices, networks, and servers can be provided according to implementation needs. In a case where data to be processed does not need to be obtained remotely, the apparatus architecture can not include a network, but only a server or a terminal device.

[0069] Figure 2 A multi-channel action recognition method based on an adaptive convolutional neural network is shown, which includes the following steps:

[0070] S1, time series data of an action collected by a plurality of sensors is obtained, and the time series data is preprocessed to obtain a plurality of sensor data.

[0071] In specific embodiments, the plurality of sensors can be a plurality of smart wearable devices, such as an acceleration sensor, a gyroscope sensor, and a gravity sensor. When a user performs an action while wearing a smart wearable device, data generated by the corresponding sensor can be collected by directly calling an Android system API. The value of the data changes over time, so it is called time series data. The overall flow framework is shown in Figure 3

[0072] In specific embodiments, the time series data in step S1 is preprocessed, specifically including:

[0073] S11, the time series data is cropped into action data of a fixed size; the size of each action data is CxT; where C is the number of sensor channels, and T is the time span of the action;

[0074] S12, the action data is processed by zero padding, denoising, and standardization, and is split according to different types of sensors to obtain a plurality of sensor data.

[0075] ​Specifically, the time series data is cropped into action data of a fixed size CxT in step S11 in a sliding window manner. For different types of actions, the time span may not be the same, so only a longest common time span needs to be defined, and the action fragments with insufficient time length are filled with zeros.

[0076] Since the sensor time segment of an action contains gravity components and a large amount of noise data, the gravity acceleration components are first subtracted on each axis of the sensor, then the data is cleaned using a smoothing mean filter to remove noise effects, and finally the data on each axis of the sensor is normalized to have a better distribution, which helps the training of the action recognition model.

[0077] The multi-source sensor data of an action is split. The action data corresponding to the same action is data from different types of sensors, such as acceleration sensor data and gyroscope sensor data. After splitting, each type of sensor (such as an acceleration sensor) corresponds to a channel of the action recognition model for extracting features of different types of sensors. The dimension of the action data is CxT.

[0078] S2, inputting the multi-sensor data into the action recognition model, wherein one type of sensor data corresponds to one channel of the action recognition model, obtaining output data of multiple channels, and the action recognition model comprises an adaptive convolutional neural network, a GRU gated recurrent neural network and a multi-head self-attention mechanism layer connected in sequence.

[0079] In a specific embodiment, taking the three axes of an acceleration sensor as an example, the structure of the action recognition model is as shown in Figure 4 The step S2 specifically comprises the following steps:

[0080] S21, obtaining input data X=input e R T×C inputting the input data into the adaptive convolutional neural network, wherein the adaptive convolutional neural network comprises a convolution operation Conv(·), a batch normalization BN(·), a Dropout(·) operation and an activation function ReLU(·); in order to extract data features in different length time segments, the size of each convolution kernel is different, and a 0 padding is performed before the convolution operation is performed on the convolution kernel of different scales, so that the output of the convolution operation of each convolution kernel is of the same dimension. The convolution operation includes a group of convolution kernels K={k1, k2, k3…k m} irepresents a convolution kernel, and length is an odd number, representing the size of the convolution kernel, where i∈{1, 2, 3…m}, m is the number of convolution kernels, and the size of each convolution kernel is k i ×C, each convolution kernel k i The corresponding convolution kernel output is P i :

[0081]

[0082] where, is the weight matrix in the convolution operation, and d o is the dimension of the channel output, and the size of each convolution kernel is k i ×C, the output dimension after the convolution operation is T×1×d o , and the dimension of 1 in the middle is eliminated, In order to facilitate subsequent calculation, d o =C.

[0083] Embodiments of the present application propose an adaptive convolutional neural network with multi-scale convolution kernels. For sensor time series, meaningful data is usually a segment rather than a single time point, so the adaptive convolutional neural network with multi-scale convolution kernels is used in the present application to extract features of different length segments on the time axis, so that the model no longer only acts on a single time point. Different length convolution kernels are used to extract features, which are respectively sent to a fully connected network to learn a weight value, and the convolution results of each convolution kernel are weighted and fused with the corresponding weight value, and finally an adaptive feature is obtained, which contains information of different length time segments, and can significantly improve the model's ability to capture time series features.

[0084] S22, merge the convolution kernel outputs P i corresponding to all convolution kernels to obtain the output result P:

[0085] P={P1, P2, P3…P m};

[0086] The output result P is passed through two fully connected layers and a softmax layer, and the importance of each convolution kernel output is learned by the model to calculate the importance score score i :

[0087] score i =softmax(FFN(P i ));

[0088]

[0089] where FFN is a two-layer fully connected layer, and is a weight matrix of the fully connected layer, and b1 and b2 are its corresponding bias terms.

[0090] S23, each output result P i is weighted and fused with its corresponding importance score score i to obtain a merged output P score =∑ i score i ×P i , where P score ∈R T×C , i∈{1,2,3…m}, and the merged output P score is then residual connected with the input data X to obtain a first-layer output result P out =X+P score , P out ∈R T×C .

[0091] S24, the first-layer output result P out ∈R T×C is split according to the dimension T to obtain P out ={M1,M2…M T}, and each M t ∈R 1×C is input to a GRU gated recurrent neural network. Three gate functions are used in a conventional LSTM: an input gate, a forget gate, and an output gate to control input values, memory values, and output values. In a GRU gated recurrent neural network, there are only two gates: an update gate and a reset gate, and their main formulas are as follows

[0092] R t =σ(W r ·[H t-1 ,M t ]);

[0093] Z t =σ(W z ·[H t-1 ,M t ]);

[0094]

[0095] The output of each unit is:

[0096]

[0097] where R t and Z t represent the reset gate and the update gate, respectively, σ is a sigmoid function, and Mt is the input at time t, H t-1 is the input of the hidden state at time t-1, W r and W z are the weight parameters of the reset gate and the update gate, respectively.

[0098] The output obtains the second layer output result H t . t ∈R 1×hidden , where hidden is the number of nodes of the hidden layer in the recurrent neural network, t = {1, 2, 3…T}.

[0099] The GRU gated recurrent neural network is used to process long-time dependence, collect features that change over time, and finally aggregate the outputs of the neural units of multiple GRU gated recurrent neural networks and send them to the Multi-Head Self-Atteneion mechanism layer, so that the model can focus on the correlation between each time node in the complete time segment from multiple angles.

[0100] S25, all the second layer output results H t are merged into a matrix H = {H1, H2, H3…H T}, H ∈R T×hidden , and the multi-head attention mechanism is implemented based on the matrix H to output the third layer output result:

[0101] Atten channel n = [head1; head2;…head h ]W O ;

[0102] Wherein, the calculation of each head is as follows:

[0103]

[0104]

[0105]

[0106]

[0107] Wherein, W O are the weight matrices of Q, K, V and Attention, respectively, j ∈ {1, 2, 3…h}; d k is the dimension, and n is the number of channels of the action recognition model.

[0108] S3, the output data of multiple channels are merged and passed through a classifier to obtain the action recognition result.

[0109] In a specific embodiment, the classifier comprises a fully connected layer and a softmax layer, and step S3 specifically comprises:

[0110] The third layer output results of each channel of the action recognition model are merged, and a fully connected layer and a softmax layer are passed through to obtain an action recognition result:

[0111] output concat =[Atten channel 1 ;Atten channel 2 ;…Atten channel n ];

[0112] output=softmax(FFN(output concat ))。

[0113] Embodiments of the present application propose a multi-channel action recognition model for processing multi-source sensor input, which separately processes different types of sensor data. The previous model combines these sensor data into a whole matrix and sends it into the model, resulting in the loss of some features that can be used to distinguish similar actions. Embodiments of the present application separate these different sensors and send them into different channels for processing, which can maximize the extraction and calculation of the features of individual sensors and time sequence correlation. The multi-head self-attention layer enables the action recognition model to focus on the correlation between each time node in the complete time segment from multiple angles. Finally, all channels are combined for calculation, which also ensures that the mutual relationship between these sensors is not lost.

[0114] In the action recognition model training phase, the collected action data is divided into a training set, a validation set, and a test set according to a ratio of 6:2:2. The action recognition model is trained through the training set, combined with the true label, and the loss function is calculated using cross-entropy (Cross-Entropy Loss). Then, the Adam optimizer is used to perform gradient backpropagation and parameter update, and the training process is repeatedly performed. The trained action recognition model is verified using the validation set, and the parameters of the classifier are adjusted, such as selecting the number of hidden units in the neural network, to optimize and determine the final action recognition model. Finally, the performance of the trained action recognition model is tested using the test set.

[0115] Finally, the trained action recognition model is used for action recognition. During the training process of the action recognition model, the action recognition model is verified using the validation set every 50 epochs, and the best model parameters are retained. After all epochs are executed, the action recognition model is tested using the test set, and the final action recognition accuracy is obtained. The action recognition model with the highest accuracy is applied to subsequent action recognition.

[0116] Further referring to Figure 5 , as an implementation of the method shown in the above figures, the present application provides an embodiment of a multi-channel action recognition device based on an adaptive convolutional neural network, which corresponds to the method embodiment shown in Figure 2 , and the device can be specifically applied to various electronic devices.

[0117] The embodiment of the present application provides a multi-channel action recognition device based on an adaptive convolutional neural network, which comprises:

[0118] A data acquisition module 1 configured to acquire time series data of an action collected by multiple sensors, and to obtain multiple sensor data by preprocessing the time series data.

[0119] A neural network module 2 configured to input the multiple sensor data into an action recognition model respectively, wherein one kind of sensor data corresponds to one channel of the action recognition model, and to obtain output data of multiple channels, the action recognition model comprising an adaptive convolutional neural network, a GRU gated recurrent neural network and a multi-head self-attention mechanism layer connected in sequence.

[0120] A classification module 3 configured to merge the output data of multiple channels, and to obtain an action recognition result through a classifier.

[0121] Reference is made to Figure 6 , which shows a structural schematic diagram of a computer device 600 of an electronic device (such as a server or terminal device shown in Figure 1 ) suitable for implementing the embodiments of the present application. Figure 6 The electronic device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0122] As shown in Figure 6 , the computer device 600 comprises a central processing unit (CPU) 601 and a graphics processor (GPU) 602, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 603 or programs loaded from a storage part 609 to a random access memory (RAM) 604. In the RAM 604, various programs and data required for the operation of the device 600 are also stored. The CPU 601, the GPU 602, the ROM 603 and the RAM 604 are connected to each other through a bus 605. An input / output (I / O) interface 606 is also connected to the bus 605.

[0123] The following components are connected to the I / O interface 606: an input part 607 including a keyboard, a mouse, etc.; an output part 608 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 609 including a hard disk, etc.; and a communication part 610 including a network interface card such as a LAN card, a modem, etc. The communication part 610 performs communication processing via a network such as the Internet. A drive 611 can also be connected to the I / O interface 606 as necessary. A removable media 612 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 611 as necessary, so that a computer program read out therefrom is installed in the storage part 609 as necessary.

[0124] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication part 610, and / or installed from the removable media 612. When the computer program is executed by the central processing unit (CPU) 601 and the graphics processor (GPU) 602, the above-described functions defined in the methods of the present application are performed.

[0125] Note that the computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a computer-readable medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present context, a computer-readable medium can be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present context, a computer-readable signal medium can include a computer-readable program code in a baseband or propagated as carrier waves in a propagated data signal associating with a carrier wave. Such a propagated signal can take a wide variety of forms including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium that can be used to carry or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The program contained in the computer-readable medium can be transmitted in any suitable format including, but not limited to, wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0126] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0127] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of apparatuses, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or in the reverse order, depending on the functionality involved. It will also be noted that each block of the block diagrams and / or flowcharts, and combinations thereof, can be implemented by dedicated hardware-based devices or by combinations of dedicated hardware-based devices and computer instructions.

[0128] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. The described modules can also be arranged in a processor.

[0129] As another aspect, the present application also provides a computer readable medium, which can be included in the electronic device described in the above embodiments, or can exist independently without being assembled into the electronic device. The computer readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: acquire time series data of an action collected by a plurality of sensors, pre-process the time series data to obtain a plurality of sensor data; input the plurality of sensor data into an action recognition model respectively, wherein one sensor data corresponds to one channel of the action recognition model, obtain output data of a plurality of channels, and the action recognition model comprises an adaptive convolutional neural network, a GRU gated recurrent neural network and a multi-head self-attention mechanism layer connected in sequence; merge the output data of the plurality of channels, and pass through a classifier to obtain an action recognition result.

[0130] The above description is merely preferred embodiments of the present application and a description of the principles of the technology used. Those skilled in the art should understand that the scope of the application involved in the present application is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or equivalent features without departing from the above inventive concept. For example, the above features can be replaced with the technical features disclosed in the present application (but not limited to) having similar functions to form technical solutions.

Claims

1. A method for multi-channel action recognition based on adaptive convolutional neural network, characterized in that, The method comprises the following steps: S1, obtaining time series data of an action collected by multiple sensors, pre-processing the time series data to obtain multiple sensor data; S2, inputting the multiple sensor data into an action recognition model respectively, wherein one kind of sensor data corresponds to one channel of the action recognition model, obtaining output data of multiple channels, and the action recognition model comprises an adaptive convolutional neural network, a GRU gated recurrent neural network and a multi-head self-attention mechanism layer connected in sequence; the adaptive convolutional neural network in the action recognition model is specifically implemented as follows: S21, the sensor data is transposed to obtain input data. Where C is the number of sensor channels and T is the time span of the action; the input data is input into the adaptive convolutional neural network, which includes a convolution operation. A batch standardization ,one Operation and an activation function The convolution operation described herein comprises a set of convolution kernels. , This represents a convolution kernel with a value of ,in m is the number of convolution kernels, and the size of each convolution kernel is . Each convolution kernel The corresponding convolution kernel output is : ； wherein, is a weight matrix in a convolution operation, , is the dimension of the channel output; S22, output all convolution kernels merge to obtain an output result P: ； The output result P is passed through two layers of fully connected layers and a layer, to obtain an importance score : ； ； where FFN is a two-layer fully connected layer, and is a weight matrix of the fully connected layer, and is its corresponding bias term. S23, each output result importance score corresponding thereto weighted fusion to obtain a merged output wherein, , the merged output is residual connected with the input data X to obtain a first layer output result ; S3, merging the output data of the multiple channels and passing through a classifier to obtain an action recognition result. 2.The method of claim 1, wherein, The time series data in the step S1 is pre-processed, and specifically includes: S11, the time series data is cut into fixed size action data in a sliding window manner; the size of each action data is ; S12, performing zero padding, denoising and standardization processing on the action data, and splitting according to different types of sensors to obtain multiple sensor data. 3.The method of claim 1, wherein, The adaptive convolutional neural network has multiple convolution kernels, and each convolution kernel has a different size. 4.The method of claim 1, wherein, The GRU gating recurrent neural network in the action recognition model of the step S2 is embodied as follows: the first layer output result is split according to the dimension T to obtain , and each is input into the GRU gating recurrent neural network to obtain a second layer output result , wherein .

5. The method of claim 4, wherein the adaptive convolutional neural network-based multi-channel action recognition method is characterized by, The multi-head self-attention mechanism layer in the action recognition model of the step S2 is specifically implemented as follows: combine all the second layer output results into a matrix , implement a multi-head attention mechanism based on the matrix H, and output a third layer output result: ； The calculation of each head is as follows: ； ； ； ； wherein, , , , are weight matrices of Q, K, V and Attention, respectively, ; is a dimension, and n is a number of channels of the action recognition model.

6. The method of claim 5, wherein the adaptive convolutional neural network-based multi-channel action recognition method is characterized by, The step S3 specifically includes: merge the third layer output results of each channel of the action recognition model, and pass through a fully connected layer and layer to obtain the action recognition result: ； 7. An apparatus for multi-channel action recognition based on an adaptive convolutional neural network, characterized by It comprises: A data acquisition module configured to obtain time series data of an action collected by multiple sensors, pre-process the time series data to obtain multiple sensor data; A neural network module configured to input the multiple sensor data into an action recognition model respectively, wherein one kind of sensor data corresponds to one channel of the action recognition model, obtain output data of multiple channels, and the action recognition model comprises an adaptive convolutional neural network, a GRU gated recurrent neural network and a multi-head self-attention mechanism layer connected in sequence; the adaptive convolutional neural network in the action recognition model is specifically implemented as follows: The sensor data is transposed to obtain the input data. Where C is the number of sensor channels and T is the time span of the action; the input data is input into the adaptive convolutional neural network, which includes a convolution operation. A batch standardization ,one Operation and an activation function The convolution operation described herein comprises a set of convolution kernels. , This represents a convolution kernel with a value of ,in m is the number of convolution kernels, and the size of each convolution kernel is . Each convolution kernel The corresponding convolution kernel output is : ； wherein, is a weight matrix in a convolution operation, , is the dimension of the channel output; output all the convolution kernels merge to obtain an output result P: ； The output result P is passed through two layers of fully connected layers and a layer, to obtain an importance score : ； ； where FFN is a two-layer fully connected layer, and is a weight matrix of the fully connected layer, and is its corresponding bias term. Each output result is multiplied by the importance score corresponding thereto to obtain a merged output wherein, , The merged output is then connected with the input data X in a residual connection manner to obtain a first-layer output result ; A classification module configured to merge the output data of the multiple channels and pass through a classifier to obtain an action recognition result.

8. An electronic device comprising: One or more processors; Storage means for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1-6.

Images