Human behavior recognition system based on multi-dimensional feature fusion and working method thereof

Abstract

The application discloses a human behavior recognition system based on multi-dimension feature fusion and a working method thereof, comprising a behavior data acquisition module, a behavior data transmission module, a behavior data storage module, a behavior data preprocessing module, a behavior feature extraction module, a behavior classification result module and a behavior recognition application module; the behavior data acquisition module, the behavior data transmission module, the behavior data preprocessing module, the behavior data storage module, the behavior feature extraction module, the behavior classification result module and the behavior recognition application module are sequentially connected. The human behavior recognition system based on multi-dimension feature fusion and the working method thereof can be applied to scenes such as prison management of prisoners, guardianship of the elderly and the like. The defects of human behavior recognition based on sensors are made up, and multi-dimension feature extraction and fusion are adopted to solve the problem of behavior recognition errors caused by inaccurate traditional feature extraction.

Description

TECHNICAL FIELD

[0001] The present application relates to an artificial intelligence technology, and in particular to a human behavior recognition system based on multi-dimensional feature fusion and a working method thereof. BACKGROUND

[0002] With the development of economic society, human behavior recognition (HAR) technology has also been well developed, such as safety monitoring, motion game, user social behavior analysis and other research fields. With the close combination of human behavior recognition and artificial intelligence and other fields, the data information obtained by behavior collection and analysis brings considerable efficiency and convenience to scientific research. Pattern recognition related to human behavior recognition has become one of the hottest research topics in the related field.

[0003] Compared with human behavior recognition technology based on wearable sensors, human behavior recognition technology based on video does not depend on the human body itself, and the collected data is more intuitive. In some specific scenarios, the endurance of wearable sensors also needs to be considered, while human behavior recognition technology based on video does not need to consider the above problems, so its technology is simpler and more reliable.

[0004] In human behavior recognition technology based on video, extracting correct and fine behavior features is the guarantee for accurately recognizing human behavior. Traditional feature extraction methods include Histogram of Oriented Gradients (HOG), Histogram of Optical Flow (HOF), Motion Energy Image (MEI), Shape Context (SC) and local binary mode (LBM) methods. Traditional feature classification methods include Artificial Neural Network (ANN), Support Vector Machine (SVM) and K-Nearest Neighbor (KNN) methods.

[0005] After 2010, with the continuous development of deep learning, deep learning methods represented by Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) have become the mainstream method of HAR technology.

[0006] However, the current HAR method still cannot extract rich and accurate behavior features, which directly leads to the error of behavior recognition result. The main reason for incomplete feature extraction is that researchers do not consider multi-dimensional feature information when extracting features. For example, CNN only extracts feature information in the spatial dimension of behavior data, without considering feature information in the time dimension. However, since a behavior is coherent in the time dimension, the feature information in the time dimension cannot be ignored.

[0007] And the collected human behavior video is usually colorful, with multiple channels. The channel is the detection of a certain feature, and the strength of the value at a certain place in the channel is the reaction to the strength of the current feature. However, when extracting behavior features, the channel dimension feature information of the video behavior data is not considered. SUMMARY

[0008] In view of the problem that the current behavior feature extraction is single-dimensional, single and rough behavior features are extracted, and the behavior recognition accuracy is not high, the present application provides a human behavior recognition system based on multi-dimensional feature fusion and a working method thereof. The multi-dimensional feature fusion network is used to extract and fuse spatial dimension features, time dimension features and channel dimension features. Compared with single-dimensional features, multi-dimensional features are richer and more accurate, so as to improve the human behavior recognition accuracy.

[0009] To achieve the above purpose, the present application provides a human behavior recognition system based on multi-dimensional feature fusion, comprising:

[0010] The behavior data acquisition module is used for real-time acquisition of monitoring video data, and different acquisition devices are used in different scenes, mainly for acquisition of the current observable behavior state of the user;

[0011] The behavior data transmission module is used for selecting a corresponding transmission mode according to the application scene for the collected video data;

[0012] The behavior data storage module is used for storing the collected original video behavior data, behavior recognition result and corresponding time stamp into the database server;

[0013] The behavior data preprocessing module is used for preprocessing the data collected by the video;

[0014] The behavior feature extraction module is used for inputting the preprocessed behavior data into the spatial and channel feature extraction unit, the convolution network basic unit and the time feature extraction unit, respectively;

[0015] The behavior classification result module is configured to fuse the multi-dimensional features extracted from the time, space and channel, calculate and predict the fused features through a full connection layer and a Softmax classifier, and finally obtain the behavior classification result.

[0016] The behavior recognition application module is configured to transmit the recognition result to an application platform for display, thereby realizing real-time monitoring and management of human behaviors.

[0017] The behavior data acquisition module, the behavior data transmission module, the behavior data preprocessing module, the behavior data storage module, the behavior feature extraction module, the behavior classification result module and the behavior recognition application module are sequentially connected.

[0018] Preferably, the behavior data acquisition module comprises a data collection device configured to collect video data and an image sensing unit configured to convert the collected optical signals into electrical signals, and the image sensing unit comprises a charge coupled device and a complementary metal oxide semiconductor.

[0019] Preferably, the behavior data transmission module comprises a wired transmission unit and a wireless transmission unit, the wired transmission unit mainly transmits baseband signals through a coaxial cable and transmits signals through an optical fiber, and the wireless transmission unit mainly transmits streaming media.

[0020] Preferably, the behavior data preprocessing module comprises:

[0021] The video data segmentation unit is configured to segment the collected long-time video data into fixed time lengths, accelerate the presentation speed of the recognition result, and make the recognition more close to real time.

[0022] The video data frame conversion unit is configured to convert the segmented video data into video frame data, so as to facilitate inputting the video frame data into the recognition model for training and recognition.

[0023] The video data denoising unit is configured to denoise the converted video frame, weaken the noise in the image data, and make the features of the image data more obvious.

[0024] The video data labeling unit is configured to label the segmented and denoised data, and perform one-hot coding on the labeled data.

[0025] Preferably, the behavior feature extraction module comprises a space and channel feature extraction unit configured to extract feature information in the space and channel dimensions, a convolution network basic unit configured to extract basic behavior feature information, and a time feature extraction unit configured to extract feature information in the time dimension.

[0026] The space and channel feature extraction unit comprises a feature pre-extraction convolutional layer, a multi-scale convolution-based space global context modeling layer and a channel feature extraction layer, the feature pre-extraction convolutional layer comprises two three-dimensional convolutional layers with a convolution kernel size of 3; the multi-scale convolution-based space global context modeling layer comprises two parallel convolutional layers with a convolution kernel size of 3 and 5 respectively and a Softmax layer for global context modeling on an image, and the channel feature extraction layer uses a squeeze-excitation module, wherein maximum pooling and average pooling are added in the squeeze operation;

[0027] The convolutional network basic unit mainly comprises five convolutional layers, and the convolutional network basic unit uses a three-dimensional convolutional network as a basic network model;

[0028] The time feature extraction unit comprises a time sequence adaptive feature extraction layer and a self-attention mechanism layer, the time sequence adaptive feature extraction layer comprises a local time sequence feature extraction branch and a global time sequence feature extraction branch, and the self-attention mechanism layer comprises a self-attention module.

[0029] The working method of the human behavior recognition system based on multi-dimensional feature fusion comprises the following steps:

[0030] Step S1: collecting video behavior information data

[0031] According to different scenes and different needs of users, video acquisition equipment and collection angles are selected, and at the same time, when collecting data, the range of human behavior appearing needs to be considered to ensure effective collection of data;

[0032] Step S2: transmitting video behavior information data

[0033] According to different scene requirements, a transmission mode is selected, and the collected original data and time stamp are transmitted to a database server for storage;

[0034] Step S3: segmenting video behavior data

[0035] The collected long time sequence video data is segmented at a fixed time length to lay a foundation for the frequency of output recognition results, and real-time human behavior recognition is realized;

[0036] Step S4: converting video behavior data into video frames

[0037] The segmented short time sequence video is converted into video frames to obtain the input form of the behavior recognition module; Step S5: video frame data denoising and label assignment

[0038] The median filtering method is used to remove noise from the converted video frame, and the 16-frame window is used to slide the data after the noise removal. In order to not lose the continuity of the action, the window slides 50% coverage each time, and then the sliding video frame is labeled and the label data is one-hot encoded;

[0039] Step S6: extracting video behavior time dimension feature information

[0040] The processed original data is input into the time dimension feature information extraction layer, and the self-attention mechanism is used to strengthen the extracted time feature;

[0041] Step S7: extracting video behavior data basic feature information

[0042] Step S8: extracting video behavior spatial and channel dimension feature information

[0043] The processed original data is first pre-extracted using convolution features, and then input into the multi-scale convolution-based context modeling and squeeze-excitation module to extract spatial features and channel features. Steps S6, S7 and S8 are performed simultaneously for feature extraction;

[0044] Step S9: feature fusion and output behavior result

[0045] The extracted time feature information, spatial feature information and channel feature information are combined and added to obtain the fused feature information. The fused information is input into the full connection layer and the Softmax layer to obtain the behavior classification result;

[0046] Step S10: determining whether an abnormal behavior occurs

[0047] The behavior classification result in step S9 is obtained, and it is determined whether it is an abnormal behavior;

[0048] Step S11: alarm

[0049] When the judgment result in step S10 is an abnormal behavior, an alarm is immediately sent to the receiving device of the guardian or manager, and the relevant personnel are notified to handle the emergency immediately. If the judgment result in step S10 is normal behavior, it will be directly displayed on the corresponding position of the system, and the system process is ended.

[0050] Preferably, the median filtering method of step S5 comprises the following methods:

[0051] For a digital signal sequence x jWhen filtering processing is performed on (-∞ < j < +∞), first, a window with a length of L is defined, L = 2N + 1, N is a positive integer; at a certain time, the signal samples in the window are x(i-N), …, x(i), …, x(i+N), wherein x(i) is the signal sample value located at the center of the window, and the L signal sample values are arranged in ascending order, and the sample value at i is defined as the output value of the median filtering, and the specific formula is as follows:

[0052] y(i) = Med[x(i-N),...,x(i),...,x(i+N)] (1)

[0053] Wherein, Med(·) represents the median filtering function, and y(i) represents the output of the median filtering.

[0054] Preferably, step S6 specifically comprises the following steps:

[0055] Step S61: extracting the time features of the video behavior data through a time sequence adaptive feature extraction layer;

[0056] Step S62: using a self-attention mechanism layer to strengthen the extracted time features.

[0057] Preferably, step S61 specifically comprises the following steps:

[0058] Step S611: the time sequence adaptive feature extraction layer focuses on learning the change pattern of the video sequence in the time sequence, so the spatial dimension of the input data X is compressed by using average pooling to obtain The time sequence adaptive feature extraction layer is composed of a local branch and a global branch, and the method is summarized as follows:

[0059]

[0060] Wherein, Y represents the extracted time sequence features, represents a convolution operator, represents an element-wise multiplication, g(·) represents a global branch, and L(·) represents a local branch.

[0061] Step S612: the learning process of the time sequence adaptive feature extraction layer is divided into a local branch and a global branch;

[0062] Wherein, the local branch aims to generate position-related importance weights using short time sequence information, and the formula is summarized as follows:

[0063]

[0064] wherein, Conv1D(*, *, *) represents a time series convolution, which accepts parameters of an input tensor, a convolution kernel size and an output channel number respectively, Sigmoid(·) represents a Sigmoid function, K is a size of an adaptive convolution kernel, and C represents a channel number;

[0065] The local branch generates is part of the time series adaptive parameter, and compared with the global branch, it is sensitive to the time series position, and therefore time series enhancement is performed in the following manner:

[0066]

[0067] wherein, is an output feature map of the local branch, F rescale (·) is a re-adjustment function; the global branch is mainly responsible for long time series modeling and capturing long time series dependence in a video, and the global branch uses a generation method of a channel time series convolution kernel, which is summarized as follows:

[0068]

[0069] wherein, represents an adaptive convolution kernel of the cth channel, K is a size of the adaptive convolution kernel, δ represents a ReLU nonlinear activation function, W1 and W2 represent feature matrices, and Softmax(·) represents a Softmax function;

[0070] Step S613: Time series adaptive aggregation of the local branch and the global branch is performed to generate a convolution kernel Θ = {Θ1, Θ2,..., Θ c in a convolution manner to learn time series structure information between video frames:

[0071]

[0072] wherein, “·” represents scalar multiplication, Z is a feature map after activation of the local branch, is a feature map obtained through time series adaptive convolution;

[0073] In step S62, the following formula is used:

[0074]

[0075] Preferably, step S8 specifically comprises the following steps:

[0076] Step S81: Two convolution layers with a convolution kernel size of 3 are used to pre-extract data features;

[0077] Step S82: A multi-scale convolution and a Softmax layer are used to model a context of data and extract spatial features;

[0078] The feature data obtained in step S81 is input into three-dimensional convolution kernels with sizes of 3 and 5 respectively, and in order to replace the convolution kernels with sizes of 3x1x1, 1x3x3 and 5x1x1, 1x5x5 respectively with less calculation, the data features after convolution are subjected to context modeling by a Softmax layer to obtain corresponding spatial features;

[0079] Step S83: extracting channel features of data by using a squeeze-excitation module

[0080] The squeeze-excitation module based on global context modeling by multi-scale convolution is obtained by combining step S82 and step S83, and the data with spatial features is input into the squeeze-excitation module to extract channel features therein, wherein not only the average pooling operation but also the maximum pooling operation is used in the squeeze operation, and the calculation method is as follows:

[0081] Squeeze operation:

[0082]

[0083] wherein, u c represents the cth output, i and j represent the height and width of the image respectively, max(·) represents the maximum pooling operation, F s (·) represents the combined operation of the maximum pooling and the average pooling.

[0084] Excitation operation:

[0085] z c =F ex (z c ,W)=σ(g(z c ,W))=σ(W2 ReLU(W1 z c )) (9)

[0086] wherein, F ex (·) represents the excitation operation, σ(·) represents the activation function, respectively represent excitation operation weight matrices.

[0087] Feature fusion:

[0088] Output=F scale (u,z c ) (10)

[0089] wherein, F scale (·) represents the feature mapping process, and Output is the output feature.

[0090] Therefore, the present application has the following beneficial effects:

[0091] 1. Practicality: the video-based behavior recognition system has higher requirements for the identification model of system transmission, etc. The system process designed by the application is more suitable for the video behavior recognition system.

[0092] 2. High reliability: compared with the existing latest identification model, the use of multi-dimensional feature fusion identification method can extract more accurate and more detailed behavior features, and obtain higher behavior recognition accuracy. The application of the identification method to the system makes the system have stronger robustness.

[0093] 3. Adaptability: in the face of different application scenarios, the application extracts features from time, space and channel of video data, and is applicable to different, complex and dynamic scenes.

[0094] The technical solutions of the application will be further described below with the help of the drawings and examples. DESCRIPTION OF DRAWINGS

[0095] Figure 1 is a structural schematic diagram of the application;

[0096] Figure 2 is a working flowchart of the application;

[0097] Figure 3 is a principle diagram of the behavior classification result module of the application;

[0098] Figure 4 is a principle diagram of the squeeze-excitation module based on multi-scale convolution for global context modeling of the application. DETAILED DESCRIPTION

[0099] The application will be further described below with the help of the drawings, and it should be noted that the embodiment takes the technical solution as the premise, and gives detailed implementation mode and specific operation process, but the protection scope of the application is not limited to the embodiment.

[0100] Figure 1 is a structural schematic diagram of the application; Figure 2 is a working flowchart of the application; Figure 3 is a principle diagram of the behavior classification result module of the application; Figure 4 is a principle diagram of the squeeze-excitation module based on multi-scale convolution for global context modeling of the application, as shown in Figures 1-4 The human body behavior recognition system based on multi-dimensional feature fusion includes:

[0101] The behavior data collection module is configured to collect monitoring video data in real time, and different collection devices are used in different scenes, and mainly collects the behavior state that can be directly observed by the user at present, and the user can adjust the collection angle position of the behavior collection device according to the own demand.

[0102] The behavior data transmission module is configured to select a corresponding transmission mode according to the application scene for the collected video data.

[0103] The behavior data storage module is configured to store the collected original video behavior data, the behavior recognition result and the corresponding time stamp into a database server.

[0104] The behavior data preprocessing module is configured to perform a preprocessing operation on the data collected by the video, and the preprocessing operation includes segmentation, conversion of video frames, denoising, allocation of behavior category labels and the like.

[0105] The behavior feature extraction module is configured to input the preprocessed behavior data into a spatial and channel feature extraction unit, a convolution network basic unit and a time feature extraction unit respectively.

[0106] The behavior classification result module is configured to fuse the multi-dimensional features extracted by the time, space and channel, calculate and predict the fused features through a full connection layer and a Softmax classifier, and finally obtain a behavior classification result.

[0107] The behavior recognition application module is configured to transmit the recognition result to an application platform to display, so as to realize real-time monitoring and management of human behaviors.

[0108] The behavior data collection module, the behavior data transmission module, the behavior data preprocessing module, the behavior data storage module, the behavior feature extraction module, the behavior classification result module and the behavior recognition application module are sequentially connected.

[0109] Preferably, the behavior data collection module includes a data collection device (such as a light source, a lens, a camera, a television device, a cloud platform, etc.) for collecting video data and an image sensing unit for converting the collected optical signal into an electrical signal, and the image sensing unit includes a charge-coupled device (CCD) and a complementary metal oxide semiconductor (CMOS). During the collection process, on the one hand, the camera facility collects the data to be collected in the form of an optical signal, and then converts the collected optical signal into an electrical signal through photoelectric sensing, and finally uses the above two image technologies to complete the conversion of video data collection.

[0110] Preferably, the behavior data transmission module comprises a wired transmission unit and a wireless transmission unit, the wired transmission unit mainly transmits baseband signals through coaxial cables and optical fiber transmission signals, and the wireless transmission unit mainly transmits streaming media, the main principle of which is to compress and encode video signals into IP streams, and a decoder is used at the data receiving end to restore the data, which increases the transmission distance of the transmitted data and reduces the transmission cost. In different scenes, specific transmission methods can be selected according to specific system needs.

[0111] Preferably, the behavior data preprocessing module comprises:

[0112] A video data segmentation unit is configured to segment the collected long-time video data into fixed time lengths, accelerate the presentation speed of the recognition result, and realize closer to real-time recognition.

[0113] A video data frame conversion unit is configured to convert the segmented video data into video frame data, so as to facilitate inputting into the recognition model for training and recognition.

[0114] A video data denoising unit is configured to denoise the converted video frame, weaken the noise in the image data, and make the image data features more obvious.

[0115] A video data label allocation unit is configured to allocate labels to the segmented and denoised data, and perform one-hot coding on the label data.

[0116] Preferably, the behavior feature extraction module comprises a spatial and channel feature extraction unit for extracting feature information in the spatial and channel dimensions, a convolutional network basic unit for extracting basic behavior feature information, and a time feature extraction unit for extracting feature information in the time dimension.

[0117] The spatial and channel feature extraction unit comprises a feature pre-extraction convolutional layer, a spatial global context modeling layer based on multi-scale convolution, and a channel feature extraction layer. The feature pre-extraction convolutional layer comprises two three-dimensional convolutional layers with a convolution kernel size of 3. The spatial global context modeling layer based on multi-scale convolution comprises two parallel convolutional layers with a convolution kernel size of 3 and 5 respectively, and a Softmax layer for global context modeling on an image. In order to reduce the amount of calculation, the 3x3x3 convolution kernel is converted into a convolution kernel with a size of 3x1x1 and 1x3x3. The convolutional layer with a convolution kernel size of 5 is processed in the same way. The channel feature extraction layer uses a squeeze-excitation module. In the squeeze operation, maximum pooling and average pooling are added. Two average pooling methods are used to find the best feature information in the average and maximum features.

[0118] The convolutional network base unit mainly consists of five convolutional layers, and the convolutional network base unit uses a three-dimensional convolutional network as a base network model.

[0119] The time feature extraction unit is composed of a time sequence adaptive feature extraction layer and a self-attention mechanism layer, the time sequence adaptive feature extraction layer includes a local time sequence feature extraction branch and a global time sequence feature extraction branch, and the self-attention mechanism layer is composed of a self-attention module.

[0120] This embodiment takes the behavior of person A in scene B as an example, and person A has an abnormal dangerous behavior in scene B. In order to more accurately and reliably identify these dangerous behaviors and prevent serious impact and harm, the behavior information is obtained through video monitoring, then the information is preprocessed to improve the reliability of the information, and finally the trained behavior recognition model is used for real-time behavior recognition and monitoring warning. Managers can set different parameters for real-time behavior recognition according to different scenes and different special personnel.

[0121] The working method of the human behavior recognition system based on multi-dimensional feature fusion comprises the following steps:

[0122] Step S1: collecting video behavior information data

[0123] According to different scenes and different needs of users, video acquisition equipment and collection angles are selected, and at the same time, when collecting data, the range of human behavior should be considered to ensure effective data collection;

[0124] Step S2: transmitting video behavior information data

[0125] According to different scene requirements, the transmission mode is selected, and the collected original data and time stamp are transmitted to the database server for storage;

[0126] Step S3: segmenting video behavior data

[0127] The collected long-time sequence video data is segmented by a fixed time length to lay the foundation for the frequency of output recognition results, realizing real-time human behavior recognition;

[0128] Step S4: converting video behavior data into video frames

[0129] The segmented short-time sequence video is converted into video frames to obtain the input form of the behavior recognition module; Step S5: video frame data denoising and label assignment

[0130] The converted video frame is filtered by a median filtering method to remove noise (median filtering has good filtering effect on impulse noise, especially in filtering noise while protecting the edge of the signal from being blurred. The algorithm of median filtering is simple and easy to implement by hardware), and the denoised video frame is taken by sliding a 16-frame window. In order to not lose the continuity of the action, the window slides with 50% coverage each time, and then the sliding video frame is labeled and the label data is one-hot encoded;

[0131] Preferably, the median filtering in step S5 specifically includes the following method:

[0132] A digital signal sequence x j When filtering a digital signal sequence x

[0133] y(i) = Med[x(i-N),...,x(i),...,x(i+N)] (1)

[0134] Where Med(·) represents the median filtering function, and y(i) represents the output of the median filtering.

[0135] Step S6: Extracting video behavior time dimension feature information

[0136] The processed original data is input into the time dimension feature information extraction layer, and the self-attention mechanism is used to strengthen the extracted time features.

[0137] Preferably, step S6 specifically includes the following steps:

[0138] Step S61: Extracting the time features of the video behavior data by a temporal adaptive feature extraction layer (Temporal Adaptive Module, TAM);

[0139] Preferably, step S61 specifically includes the following steps:

[0140] Step S611: TAM focuses on learning the change pattern of the video sequence in time sequence, so average pooling is used to compress the spatial dimension of the input data X to obtain TAM is composed of local branches and global branches, and the method is summarized as follows:

[0141]

[0142] where Y represents the extracted timing feature, denotes the convolution operator, denotes the element-wise multiplication, g(·) represents the global branch, and L(·) represents the local branch.

[0143] Step S612: decompose the learning process of the TAM into a local branch and a global branch;

[0144] where the local branch aims to generate a position-dependent importance weight using short timing information, and the formula is summarized as follows:

[0145]

[0146] where Conv1D(*,*,*) represents a timing convolution, which accepts parameters including an input tensor, a convolution kernel size, and an output channel number, Sigmoid(·) represents a Sigmoid function, K is the size of an adaptive convolution kernel, and C represents a channel number;

[0147] The local branch generates an importance weight is part of the timing adaptive parameter, and compared with the global branch, it is sensitive to the timing position, and therefore timing enhancement is performed in the following manner:

[0148]

[0149] wherein, is the output feature map of the local branch, and F rescale (·) is a readjustment function.

[0150] The global branch is the core of the TAM, which generates a video-dependent adaptive convolution kernel based on global timing information. The global branch is mainly responsible for long timing modeling and captures long timing dependencies in the video. The global branch uses a channel timing convolution kernel generation method, which is summarized as follows:

[0151]

[0152] wherein, represents an adaptive convolution kernel of the cth channel, K is the size of the adaptive convolution kernel, δ represents a ReLU nonlinear activation function, W1 and W2 represent feature matrices, and Softmax(·) represents a Softmax function.

[0153] Step S613: perform timing adaptive aggregation of the local branch and the global branch to generate a convolution kernel Θ = {Θ1, Θ2, · · ·, Θ c} in a convolution manner to learn the timing structure information between video frames:

[0154]

[0155] wherein “·” represents scalar multiplication, Z is a feature map after local branch activation, is a feature map obtained through timing adaptive convolution;

[0156] Step S62: using a self-attention mechanism layer to strengthen the extracted time features.

[0157] In step S62, the following formula is used:

[0158]

[0159] Step S7: extracting video behavior data basic feature information;

[0160] The processed original data is input into a basic network model to obtain the basic features of the data, wherein the basic network model uses a C3D network, the network has 8 convolution layers, the convolution kernel size is set to 3, and the step size is set to 1; in 5 pooling layers, except for the first filter size, the filter size is set to 2, and the step size is set to 2; two FC layers with a unit number of 4096 and a Softmax classification layer;

[0161] The data size input into the behavior recognition model is batch_size×channel×clip_len×height×width, and the output size of the Softmax layer is batch_size×n, wherein batch_size is the training batch size, channel is the number of image channels, clip_len is the number of video frames contained in a sliding window, height is the height of the video frame, width is the width of the video frame, and n is the number of human behavior recognition categories.

[0162] Step S8: extracting video spatial and channel dimension feature information

[0163] The processed original data is first pre-extracted using a convolution feature, and then input into a multi-scale convolution-based context modeling and squeeze-excitation module to extract spatial features and channel features, wherein steps S6, S7 and S8 are performed simultaneously.

[0164] Preferably, step S8 specifically includes the following steps:

[0165] Step S81: using two convolution layers with a convolution kernel size of 3 to pre-extract data features;

[0166] Step S82: using multi-scale convolution and a Softmax layer to model the context of the data and extract spatial features;

[0167] The feature data obtained in step S81 is input into three-dimensional convolution kernels with sizes of 3 and 5 respectively, and in order to replace the convolution kernels with sizes of 3x1x1 and 1x3x3 and 5x1x1 and 1x5x5 respectively with less calculation, the data features after convolution are subjected to context modeling by a Softmax layer to obtain corresponding spatial features;

[0168] Step S83: Extracting channel features of data using a squeeze-excitation module

[0169] Combining step S82 and step S83, a squeeze-excitation module based on multi-scale convolution for global context modeling (MGSE) is obtained, and the data with spatial features is input into the squeeze-excitation module to extract channel features therein, wherein not only an average pooling operation but also a maximum pooling operation is used in the squeeze operation, and the calculation method is as follows:

[0170] Squeeze operation:

[0171]

[0172] wherein, u c represents the cth output, i and j represent the height and width of the image respectively, max(·) represents the maximum pooling operation, F s (·) represents the combined operation of the maximum pooling and the average pooling;

[0173] Excitation operation:

[0174] z c =F ex (z c ,W)=σ(g(z c ,W))=σ(W2 ReLU(W1 z c )) (9)

[0175] wherein, F ex (·) represents the excitation operation, σ(·) represents the activation function, and respectively represent the excitation operation weight matrix;

[0176] Feature fusion:

[0177] Output=F scale (u,z c ) (10)

[0178] wherein, F scale(·) represents a process of feature mapping, and Output is an output feature.

[0179] Step S9: feature fusion and output of behavior result

[0180] The extracted time feature information, spatial feature information and channel feature information are added to obtain fused feature information, and the fused information is input into a full connection layer and a Softmax layer to obtain a behavior classification result.

[0181] Step S10: judging whether an abnormal behavior occurs

[0182] The behavior classification result in step S9 is obtained, and it is judged whether it is an abnormal behavior.

[0183] Step S11: alarming

[0184] When the judgment result in step S10 is an abnormal behavior, an alarm is immediately sent to a receiving device of a guardian or a manager, and relevant personnel are notified to immediately handle the emergency; if the judgment result in step S10 is a normal behavior, it is directly displayed on a corresponding position of the system, and the system process is ended.

[0185] Therefore, the human behavior recognition system based on multi-dimensional feature fusion is used, data collection devices (for example, a camera, a smart phone, etc.) are used to collect the data to be collected in the form of optical signals, photoelectric sensing is used to convert the collected optical signals into electrical signals, and conversion of video acquisition data is completed. After data collection is completed, the data converted into electrical signals enters a data transmission stage. Data transmission is divided into wired transmission and wireless transmission, and a suitable transmission mode can be selected according to a specific scene. The received video data is segmented, converted into video frames, and subjected to pretreatment operations such as denoising. The pretreated video data is input into a behavior recognition classification model to perform behavior recognition, and a behavior recognition result is obtained. The behavior recognition result is displayed on an application platform. The collected original video and the last behavior recognition result are stored on a data server together with a time stamp.

[0186] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application but not to limit them, although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can still be modified or replaced by equivalents, and these modifications or equivalent replacements cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present application.

Claims

1. A human behavior recognition system based on multi-dimensional feature fusion, characterized in that: include: Behavioral data acquisition module: used to collect monitoring video data in real time. Different acquisition devices are used in different scenarios. It mainly collects the user's current behavioral status that can be directly observed. Behavioral data transmission module: used to select the appropriate transmission method for the acquired video data according to the application scenario; Behavior data storage module: used to store the collected raw video behavior data, behavior recognition results and corresponding timestamps into the database server; Behavioral data preprocessing module: used to preprocess the data captured from the video. The behavioral feature extraction module is used to input the preprocessed behavioral data into the spatial and channel feature extraction unit, the convolutional network basic unit, and the temporal feature extraction unit, respectively. The spatial and channel feature extraction unit is used to extract feature information in the spatial channel dimension; the convolutional network basic unit is used to extract basic behavioral feature information; and the temporal feature extraction unit is used to extract feature information in the temporal channel dimension. Behavior classification result module: It is used to fuse behavioral feature information as well as feature information in the time channel dimension and the spatial channel dimension. The fused features are calculated and predicted through a fully connected layer and a Softmax classifier to finally obtain the behavior classification result. And the behavior recognition application module: used to transmit the recognition results to the application platform for display, thereby realizing real-time monitoring and management of human behavior; The behavior data acquisition module, behavior data transmission module, behavior data preprocessing module, behavior data storage module, behavior feature extraction module, behavior classification result module, and behavior recognition application module are connected in sequence.

2. The human behavior recognition system based on multi-dimensional feature fusion according to claim 1, characterized in that: The behavior data acquisition module includes a data collection device for collecting video data and an image sensing unit for converting the collected optical signals into electrical signals. The image sensing unit includes a charge-coupled device and a complementary metal-oxide-semiconductor. 3.The human body behavior recognition system based on multi-dimensional feature fusion according to claim 1, characterized in that: The behavioral data transmission module includes a wired transmission unit and a wireless transmission unit. The wired transmission unit mainly transmits baseband signals via coaxial cable and signals via optical fiber, while the wireless transmission unit mainly transmits streaming media.

4. The human behavior recognition system based on multi-dimensional feature fusion according to claim 1, characterized in that: The behavioral data preprocessing module includes: The video data segmentation unit is used to segment the acquired long-time video data into fixed-duration segments to speed up the presentation of recognition results and make it closer to real-time recognition. The video data to frame conversion unit is used to convert the segmented video data into video frame data, which is convenient to be input into the recognition model for training and recognition. The video data denoising unit is used to denoise the converted video frames, reduce noise in the image data, and make the image data features more obvious. It also includes a video tag assignment unit, which is used to assign tags to the segmented and denoised data and perform one-hot encoding on the tag data.

5. The human behavior recognition system based on multi-dimensional feature fusion according to claim 1, characterized in that: The space and channel feature extraction unit comprises a feature pre-extraction convolutional layer, a multi-scale convolution-based space global context modeling layer and a channel feature extraction layer, the feature pre-extraction convolutional layer comprises two three-dimensional convolutional layers with a convolution kernel size of 3; the multi-scale convolution-based space global context modeling layer comprises two parallel convolutional layers with a convolution kernel size of 3 and 5 respectively and a Softmax layer for global context modeling on an image, and the channel feature extraction layer uses a squeeze-excitation module, wherein maximum pooling and average pooling are added in the squeeze operation; The convolutional network basic unit mainly comprises five convolutional layers, and the convolutional network basic unit uses a three-dimensional convolutional network as a basic network model; The time feature extraction unit is composed of a time sequence adaptive feature extraction layer and a self-attention mechanism layer, the time sequence adaptive feature extraction layer comprises a local time sequence feature extraction branch and a global time sequence feature extraction branch, and the self-attention mechanism layer is composed of a self-attention module.

6. The working method of the human behavior recognition system based on multi-dimensional feature fusion according to any one of claims 1-5, characterized in that: The method comprises the following steps: Step S1: collecting video behavior information data According to different scenes and user needs, select video acquisition equipment and collection angle, at the same time, when collecting data, the range of human behavior should be considered to ensure effective data collection; Step S2: transmitting video behavior information data According to different scene needs, select transmission mode, and transmit to database server to store collected original data and time stamp; Step S3: segmenting video behavior data The collected long time sequence video data is segmented by fixed time length to lay the foundation for the frequency of output recognition result, and real-time human behavior recognition is realized; Step S4: converting video behavior data into video frames The segmented short time sequence video is converted into video frames to obtain the input form of the behavior recognition module; Step S5: video frame data denoising and label assignment The converted video frames are denoised using median filtering method, and the denoised video frames are taken data using a 16-frame window sliding, in order not to lose the continuity of the action, the window sliding has a 50% coverage rate each time, then the sliding video frames are labeled, and the label data is one-hot encoded; Step S6: extracting feature information in the time channel dimension of video behavior The processed original data are input into the time dimension feature information extraction layer, and the self-attention mechanism is used to strengthen the extracted time features; Step S7: extracting video behavior feature information Step S8: extracting feature information in the space channel dimension of video behavior The processed original data are first pre-extracted using convolutional features, and then input into the multi-scale convolution-based context modeling and squeeze-excitation module to extract space features and channel features, wherein steps S6, S7 and S8 extract features at the same time; Step S9: feature fusion and output of behavior result The extracted behavior feature information, feature information in the time channel dimension and feature information in the space channel dimension are added to obtain fused feature information, and the fused information is input into a full connection layer and a Softmax layer to obtain a behavior classification result; Step S10: judging whether abnormal behavior occurs The behavior classification result in step S9 is obtained, and it is judged whether it is abnormal behavior; Step S11: alarm When the judgment result in step S10 is abnormal behavior, an alarm is immediately sent to the receiving device of the guardian or manager, and the relevant personnel are notified to handle the emergency immediately; if the judgment result in step S10 is normal behavior, it will be directly displayed on the corresponding position of the system, and the system process is ended.

7. The working method of the human behavior recognition system based on multi-dimensional feature fusion according to claim 6, characterized in that: The median filtering in step S5 specifically includes the following method: A digital signal sequence x j When filtering a digital signal sequence x (j) (j = -∞, ∞), a window of length L with odd number L = 2N + 1, N is a positive integer, is defined first. Suppose at a certain time, the signal samples in the window are x(i-N), …, x(i), …, x(i+N), where x(i) is the signal sample value located at the center of the window, and the L signal sample values are arranged in ascending order, where the sample value at i is defined as the output value of the median filtering, and the specific formula is as follows: y(i)=Med[x(i-N),...,x(i),...,x(i+N)] (1) Where Med(·) represents the median filtering function, and y(i) represents the output of the median filtering. 8.The working method of the human behavior recognition system based on multi-dimensional feature fusion according to claim 7, characterized in that: Step S6 specifically includes the following steps: Step S61: extracting the time characteristics of the video behavior data through the time sequence adaptive feature extraction layer; Step S62: using the self-attention mechanism layer to strengthen the extracted time characteristics. 9.The working method of the human behavior recognition system based on multi-dimensional feature fusion according to claim 8, characterized in that: Step S61 specifically includes the following steps: Step S611: The time-adaptive feature extraction layer focuses on learning the change pattern of the video sequence in time sequence, so the spatial dimension of the input data X is compressed by using average pooling to obtain The time-adaptive feature extraction layer is composed of a local branch and a global branch, and the method is summarized as follows: wherein Y represents the extracted timing feature, denotes a convolution operator, denotes an element-wise multiplication, g(·) denotes a global branch, and L(·) denotes a local branch. Step S612: decomposing the learning process of the time sequence adaptive feature extraction layer into local branches and global branches Where the local branch aims to generate position-related importance weights using short time sequence information, and the formula is summarized as follows: Where Conv1D(*,*,*) represents a time sequence convolution, which accepts input tensor, convolution kernel size and output channel number as parameters, Sigmoid(·) represents the Sigmoid function, K is the size of the adaptive convolution kernel, and C represents the number of channels; locally branched is part of the timing adaptation parameters and is characterized by a sensitivity to the timing position compared to the global branch, so that the timing enhancement is performed in the following way: wherein, is the output feature map of the local branch, F rescale (·) is a re-adjustment function; the global branch is mainly responsible for long-term modeling, capturing long-term dependencies in the video, and the global branch uses a generation method of a channel time convolution kernel, which is summarized as follows: wherein, denotes an adaptive convolution kernel for the c-th channel, K is the size of the adaptive convolution kernel, δ denotes a ReLU non-linear activation function, W1and W2represent feature matrices, and Softmax(·) denotes a Softmax function; Step S613: Time sequence adaptive aggregation of local branches and global branches, and the generated convolution kernel Θ = {Θ1, Θ2, ···, Θ c} learns the time sequence structure information between video frames in a convolution manner: wherein "·" represents scalar multiplication, Z is a feature map after local branch activation, is a feature map obtained through a timing adaptive convolution; The following formula is used in step S62: 10.The working method of the human behavior recognition system based on multi-dimensional feature fusion according to claim 9, characterized in that: Step S8 specifically includes the following steps: Step S81: using two convolution layers with a convolution kernel size of 3 to pre-extract data features; Step S82: using multi-scale convolution and Softmax layer to model the context of the data and extract spatial features; The feature data obtained in step S81 is input into three-dimensional convolution layers with a convolution kernel size of 3 and 5, respectively, in order to reduce the amount of calculation, convolution kernel sizes of 3x1x1 and 1x3x3 are used instead, and after convolution, the data features are modeled by Softmax layer to obtain corresponding spatial features; Step S83: using the squeeze-excitation module to extract the channel features of the data The squeeze-excitation module based on multi-scale convolution for global context modeling is obtained by combining step S82 and step S83, and the data with spatial features is input into the squeeze-excitation module to extract the channel features therein, wherein not only the average pooling operation but also the maximum pooling operation is used in the squeeze operation, and the calculation method is as follows: Squeeze operation: where u c represents the c-th output, i and j represent the height and width of the image respectively, max(·) represents the max-pooling operation, F s (·) represents the combination operation of max-pooling and average-pooling; Excitation operation: z s = F ex (z c , W) = σ(g(z c , W)) = σ(W2ReLU(W1z c )) (9) wherein F ex (·) represents an excitation operation, and σ(·) represents an activation function, respectively represent excitation operation weight matrices; Feature fusion: Output = F scale (u, z s ) (10) where F scale (·) denotes the process of feature mapping, and Output is the output feature.

Images