A real-time behavior recognition system, method, device and storage medium based on multi-source multi-modal data and multi-dimensional convolution model

Abstract

The application relates to a real-time behavior recognition system, method, device and storage medium based on multi-source multi-modal data and a multi-dimensional convolution model, which comprises a behavior information physical layer, a behavior information access layer, a behavior information platform layer and a behavior information application layer.The transmission of the behavior information access layer of the application adopts a wireless network to realize the transmission of behavior information; a multi-dimensional convolution model is adopted to greatly improve the precision in the face of high dynamic and complex intelligent scenes; and the use of a filtering algorithm makes the system have strong robustness.The application can identify the behavior categories of users in real time, can be used for the monitoring and management of special personnel, and can be used in the fields of athlete rehabilitation management and smart home, etc.

Description

TECHNICAL FIELD

[0001] The present application relates to a real-time behavior recognition system, method, device and storage medium based on multi-source multi-modal data and multi-dimensional convolution model, belonging to the field of artificial intelligence. BACKGROUND

[0002] With the development of deep learning technology and the continuous improvement of computer data processing capability, artificial intelligence technology has developed rapidly. Computer vision and various wearable devices, as part of the artificial intelligence system, are gradually integrated into everyone's daily life. Various intelligent devices are widely used in industrial production, transportation and even military fields.

[0003] Human behavior recognition is a research hotspot in the field of artificial intelligence. A human behavior recognition system processes and analyzes the perceived behavior information using a reasonable algorithm model to achieve accurate recognition of human behavior.

[0004] The behavior information processed by the human behavior recognition system mainly includes two types: the first type is visual-based behavior information collected by a camera device. The second type is sensor-based behavior information collected by intelligent hardware.

[0005] Currently, the mainstream behavior recognition technology mainly uses visual-based behavior information. However, in practical applications, relying solely on video may not accurately identify behavior types due to blind spots and insufficient camera light. The sensor-based recognition scheme is more simple and flexible, and can make up for the shortcomings of the video recognition scheme.

[0006] The behavior recognition mainly adopts machine learning and deep learning algorithms. At present, the mainstream machine learning algorithms include K-Nearest Neighbor (KNN), Support Vector Machine (SVM), Random Forest (RF), etc. Such pixel-based methods rarely consider the spatial structure information between adjacent pixels, fail to fully utilize the spatial features of behavior information, and the classification results are easily disturbed by noise. The mainstream deep learning algorithms mainly include Deep Neural Network (DNN), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Deep Belief Network (DBN), etc. Compared with the classification technology based on traditional machine learning, the deep learning technology can overcome the influence of sensor noise to a certain extent. However, such methods cannot fully learn the information and rules of similar behaviors. For example, the one-dimensional convolution layer only focuses on the time characteristics of behaviors, and cannot consider the spatial characteristics, so it will cause misjudgment to a certain extent and reduce the accuracy of behavior recognition.

[0007] In the face of high dynamic and complex intelligent scenes, the flexibility and adaptability of the system are poor due to the inherent disadvantages of a single sensor. Therefore, in order to improve the accuracy of behavior recognition and the reliability of the system, the technology of fusing multi-source and multi-modal sensor data for behavior recognition has more advantages. SUMMARY

[0008] In view of the current single source of behavior information acquisition and high misjudgment of the prediction model, and considering the needs of special scenes, the application provides a real-time behavior recognition system based on multi-source and multi-modal data and multi-dimensional convolution model.

[0009] The multi-source and multi-modal sensor data used in the application is collected by the Inertial Measurement Unit (IMU) sensor worn on different positions of the human body. Multi-source means that multiple IMU sensors are collected simultaneously, and multi-modal means that the IMU sensor includes an accelerometer and a gyroscope, which can collect data of two modalities of acceleration and angular velocity, and can be extended to multiple modalities.

[0010] The application adopts a multidimensional convolution (MDC) network model, and is matched with one-dimensional, two-dimensional and three-dimensional convolution structures, has better adaptability to data containing space-time characteristics than a two-dimensional convolution neural network, can extract deeper spatial and temporal features of data, and thus better recognition effect is obtained.

[0011] The application further provides a real-time behavior recognition method based on multi-source multi-modal data and a multidimensional convolution model, a computer device and a storage medium. SUMMARY

[0013] A real-time behavior recognition system based on multi-source multi-modal data and a multidimensional convolution model comprises a behavior information physical layer, a behavior information access layer, a behavior information platform layer and a behavior information application layer which are connected in sequence.

[0014] The application provides a feasible scheme for real-time behavior recognition based on sensors, makes up for defects of video-based behavior recognition, and greatly improves the accuracy of behavior recognition by using multi-source multi-modal sensor data compared with single-modal sensor data.

[0015] The technical scheme of the application is as follows:

[0016] A real-time behavior recognition system based on multi-source multi-modal data and a multidimensional convolution model comprises a behavior information physical layer, a behavior information access layer, a behavior information platform layer and a behavior information application layer which are connected in sequence.

[0017] The behavior information physical layer is used for collecting behavior information of a user, and the behavior information comprises X, Y and Z three-axis data of an acceleration sensor and X, Y and Z three-axis data of an angular velocity sensor.

[0018] The behavior information access layer is used for transmitting the collected behavior information to the behavior information access layer.

[0019] The behavior information platform layer is used for preprocessing behavior information, specifically filtering, normalization, multi-source multi-modal sensor data fusion and time series-based interception in sequence, inputting the intercepted behavior data into the built behavior recognition model, and storing the output behavior recognition result and the original behavior information in real time; the multi-dimensional convolution model is a behavior recognition model.

[0020] Normalization refers to mean normalization processing of numerical data in the behavior information, that is, transforming the values of all similar sensors to [-1, 1];

[0021] Multi-source multi-modal sensor data fusion refers to aligning the behavior information carried by different sensors according to timestamps and then fusing them;

[0022] Time series-based interception refers to segmenting the continuous time series collected by the sensor through a sliding window method to ensure that a complete motion data falls within a sliding window.

[0023] The behavior information application layer is used for displaying the behavior recognition result output by the behavior recognition model.

[0024] According to the application, the behavior information physical layer comprises a behavior information collection module, and the behavior information collection module comprises a plurality of sensor modules; the sensor modules adopt inertial measurement unit sensors to collect X, Y and Z three-axis data of an acceleration sensor and X, Y and Z three-axis data of an angular velocity sensor.

[0025] According to the application, the behavior information access layer comprises a behavior information transmission module, and the behavior information transmission module adopts a wireless network to transmit the behavior information collected by the behavior information collection module.

[0026] Further preferably, the behavior information transmission module adopts any one of WiFi, Bluetooth, ZigBee, 4G, 5G and Long Range Radio (LoRa) transmission technology to transmit the behavior information collected by the behavior information collection module.

[0027] According to the application, the behavior information platform layer comprises a server module, a behavior information preprocessing module and an MDC network module.

[0028] The behavior information preprocessing module is configured to preprocess the behavior information, and includes a behavior information denoising unit, a behavior information normalization unit, a behavior information fusion unit, and a behavior information sliding window unit. The behavior information transmission module transmits the behavior information to the behavior information denoising unit. The behavior information denoising unit performs denoising processing on the behavior information by using a wavelet threshold method, and inputs the behavior information to the behavior information normalization unit for processing by using a mean normalization method. The behavior information fusion unit aligns and fuses the normalized behavior information of the multi-source multi-modal sensors according to timestamps, and inputs the behavior information to the behavior information sliding window unit for intercepting by adjusting the size, cutting frequency, and step length of the sliding window.

[0029] The MDC network module includes a data remodeling unit, a one-dimensional convolution network unit, a two-dimensional convolution network unit, a three-dimensional convolution network unit, a feature fusion unit, a multi-head attention unit, and an output unit. The data remodeling unit adjusts the dimension of the behavior information output by the behavior information preprocessing module, divides the behavior information according to sensor types and wearing positions, carries spatial information in the original one-dimensional time sequence, remodels the original one-dimensional behavior data into two-dimensional and three-dimensional behavior data, and inputs the one-dimensional, two-dimensional, and three-dimensional behavior data to the one-dimensional convolution network unit, the two-dimensional convolution network unit, and the three-dimensional convolution network unit, respectively. The one-dimensional convolution network unit includes three one-dimensional convolution kernels for extracting the time characteristics of the behavior information. The two-dimensional convolution network unit includes three two-dimensional convolution kernels for extracting the spatial characteristics of the behavior information. The three-dimensional convolution network unit includes three three-dimensional convolution kernels for extracting deep spatial characteristics of the behavior information. The extracted time characteristics, spatial characteristics, and deep spatial characteristics of the behavior information are fused by the feature fusion unit, input to the multi-head attention unit, analyzed for the relationship between the time sequence and the space, and analyzed for global dependency. The obtained data is input to the output unit, sequentially passes through an unfolding layer, a fully connected layer, and a Softmax classifier, is operated on all feature data, and identifies the current behavior.

[0030] The server module includes a database server unit configured to store original behavior information and behavior recognition results.

[0031] According to the application, the behavior information application layer includes a behavior and position information display module. The behavior and position information display module is connected to the output unit in the MDC network module, and is configured to display the behavior recognition result output by the output unit.

[0032] A real-time behavior recognition method based on multi-source multi-modal data and a multi-dimensional convolution model is realized by using the real-time behavior recognition system based on multi-source multi-modal data and a multi-dimensional convolution model, and includes the following steps:

[0033] Step S1: Behavioral Information Collection

[0034] Collect user behavior information, including X, Y, and Z axis data from the accelerometer and X, Y, and Z axis data from the angular velocity sensor;

[0035] Step S2: Behavioral Information Transmission

[0036] The collected behavioral information is transmitted to a local server or a cloud server;

[0037] Step S3: Behavioral Information Preprocessing

[0038] Preprocessing behavioral information specifically refers to: sequentially performing filtering, normalization, multi-source multimodal sensor data fusion, and time-series-based data extraction;

[0039] Step S4: Build and train the behavior recognition model

[0040] The behavior recognition model, namely the MDC network module, includes a data reshaping unit, a one-dimensional convolutional network unit, a two-dimensional convolutional network unit, a three-dimensional convolutional network unit, a feature fusion unit, a multi-head attention unit, and an output unit;

[0041] The parameters set by the user include: the shape of the input data, the shape of the reshaped data, and the number of convolutional network units. Through several iterations of training, the parameters of the behavior recognition model are continuously optimized, and finally the trained behavior recognition model is obtained.

[0042] Step S5: Real-time recognition of behavioral information

[0043] Real-time collected and preprocessed behavioral information is input into a pre-trained behavior recognition model to perform real-time recognition of the current behavior and output the classification result corresponding to the behavioral information.

[0044] Step S6: Display of Behavioral Information

[0045] The behavior recognition results are displayed through the behavior information display module.

[0046] Further preferred, in step S4, the learning rate is set to 0.0005, and the cross-entropy loss function is used. Through iterative training, the loss function value continuously decreases, and the parameters of the behavior recognition model are further optimized, finally obtaining the trained behavior recognition model. The loss function is shown in formula (1):

[0047]

[0048] Where A is the number of samples, M is the number of categories, and y ij The sign function is p, which takes the value 1 if the true class of sample i is equal to j, and 0 otherwise.ij to observe the prediction probability of the sample i belonging to the category j.

[0049] According to the application, preferably, step S1 refers to wearing the sensor at different positions of the user to obtain the sensor data, i.e. acceleration and angular velocity data.

[0050] According to the application, preferably, step S3 comprises the following steps:

[0051] Step S31: denoising of behavior information

[0052] The behavior information is denoised by the wavelet threshold method.

[0053] Step S32: normalization of behavior information

[0054] The mean normalization method is used for mean normalization processing, so that features of different dimensions are in the same numerical order of magnitude.

[0055] Step S33: fusion of multi-source multi-modal sensor data

[0056] The behavior information carried by the sensors worn at different positions is aligned according to the time stamp.

[0057] Step S34: time series-based truncation

[0058] Referring to the parameters preset by the user, including the size of the sliding window, the cutting frequency and the step length of the sliding window, the behavior information processed in step S33 is subjected to sliding window processing, so that the behavior information is input into the MDC network module in the form of information blocks.

[0059] Further preferably, in step S31, the behavior information is denoised by the wavelet threshold method, specifically:

[0060] Suppose the actual measurement signal is: f(t)=s(t)+e(t), t=1,2,…,N, s(t) is the original signal, f(t) is the noisy signal, e(t) is the Gaussian white noise, e(t)~N(0,σ 2 ), σ is the noise intensity, the denoising process is to remove the noise e(t) from the signal f(t) to obtain the best approximation of the original signal s(t);

[0061] First, it is discretely sampled to obtain an N' point discrete signal f(x), x=0,1,2,…,N'-1, and the wavelet transform coefficient is as shown in formula (2):

[0062]

[0063] where W f (j,k) is the wavelet coefficient, ψ(2 -jx-k) is a scaling function, j is a scale parameter, k is a unit number of scaling function translation, and the recursive implementation method of formula (2) is obtained through double scaling equations (3) and (4):

[0064] S f (j+1,k)=S f (j,k)*h(j,k) (3)

[0065] W f (j+1,k)=S f (j,k)*g(j,k) (4)

[0066] Wherein, the symbol "*" represents convolution, h and g represent low-pass and high-pass filters respectively, S f (0,k) represents the original signal f(k), S f (j,k) represents the approximation coefficient at the j scale, and then the wavelet transform reconstruction formula is shown in formula (5):

[0067] S f (j-1,k)=S f (j,k)*h(j,k)+W f (j,k)*g(j,k) (5)

[0068] Secondly, the threshold value is determined by using a general threshold rule, and the threshold value algorithm formula is shown in formula (6):

[0069]

[0070] Wherein, sigma = MAD / 0.6745, MAD is the median value of the absolute value of the first layer wavelet decomposition coefficient, 0.6745 is the adjustment coefficient of the standard deviation of Gaussian noise, and L is the size or length of the signal;

[0071] Then, the wavelet transform coefficients of the measurement signal are processed by a nonlinear threshold, for each layer of high-frequency coefficients from the first layer to the Vth layer, a soft threshold function is used for processing, the absolute value of the signal is compared with the threshold value, the points less than the threshold value are set to zero, and the points greater than or equal to the threshold value are shrunk to zero, becoming the difference between the point value and the threshold value, and the low-frequency coefficients of each layer are not processed; the soft threshold function is shown in formula (7):

[0072]

[0073] Wherein, W j,k is the simplified form of W f (j,k), and T is a threshold value parameter greater than zero.

[0074] Finally, according to the low-frequency coefficient of the Nth layer of wavelet decomposition and the high-frequency coefficient of the 1st layer to the Vth layer after quantization, the wavelet reconstruction of the signal is performed according to formula (4) to obtain the denoised signal.

[0075] According to the application, the shape of the input data, the size of the one-dimensional convolution network unit convolution kernel, the size of the two-dimensional convolution network unit convolution kernel, the size of the three-dimensional convolution network unit convolution kernel, the number of one-dimensional convolution network units, the number of two-dimensional convolution network units, the number of three-dimensional convolution network units, and the number of attention heads of the multi-head attention unit are set in step S4.

[0076] The two-dimensional data reshaping parameters x, y, z and the three-dimensional data reshaping parameters r, s, t, u of the data reshaping unit are set; x, y, z are the length, width, and channel number of the reshaped two-dimensional data, respectively; r, s, t, u are the length, width, height, and channel number of the reshaped three-dimensional data, respectively.

[0077] Batch_Size and Window_Size are set; Batch_Size refers to the number of behavior samples in each batch in the MDC network module, and Window_Size refers to the length of the sample data.

[0078] The size of the convolution kernel of the first convolution layer in the i-th one-dimensional convolution network unit is set to 1, and the number is o1 i The size of the convolution kernel of the second convolution layer is set to 3, and the number is p1 i The size of the convolution kernel of the third convolution layer is set to 1, and the number is q1 i The padding mode of the three convolution layers is "SAME".

[0079] The size of the convolution kernel of the first convolution layer in the i-th two-dimensional convolution network unit is set to 1x1, and the number is o2 i The size of the convolution kernel of the second convolution layer is set to 3x3, and the number is p2 i The size of the convolution kernel of the third convolution layer is set to 1x1, and the number is q2 i The padding mode of the three convolution layers is "SAME".

[0080] The size of the convolution kernel of the first convolution layer in the i-th three-dimensional convolution network unit is set to 1x1x1, and the number is o3 i The size of the convolution kernel of the second convolution layer is set to 3x3x3, and the number is p3 i The size of the convolution kernel of the third convolution layer is set to 1x1x1, and the number is q3 i The padding mode of the three convolution layers is "SAME".

[0081] The number of one-dimensional convolution network units, two-dimensional convolution network units, and three-dimensional convolution network units is set to c.

[0082] Setting feature fusion unit shaping parameters

[0083] Setting the full connection output length in the output unit, i.e., the number of classifications, as Output_Length;

[0084] Comprising the following steps:

[0085] Step S41: Assuming that the input data is composed of sensor data fusion of n inertial measurement units, the behavior information with a size of Batch_Size×Window_Size×6n, Window_Size is the length of one-dimensional data, and 6n is the number of data channels;

[0086] Step S42: After the behavior information with a size of Batch_Size×Window_Size×6n passes through the data reshaping unit, two forms of Batch_Size×Window_Size×3×2n and Batch_Size×Window_Size×2×3×n are generated, 3×2n represents that 2n sensor modules each carry 3-axis behavior data, and 2n sensor units are divided into n accelerometers and n gyroscopes, and 2×3×n represents that n inertial measurement units each have an accelerometer and a gyroscope, and the data of each sensor is 3-axis data;

[0087] Step S43: The behavior information with a size of Batch_Size×Window_Size×6n, Batch_Size×Window_Size×3×2n, and Batch_Size×Window_Size×2×3×n is respectively input into a one-dimensional convolution network unit, a two-dimensional convolution network unit, and a three-dimensional convolution network unit, the one-dimensional convolution network unit, the two-dimensional convolution network unit, and the three-dimensional convolution network unit all adopt a residual structure, a residual block includes a direct mapping part and a residual part, and in the two-dimensional convolution network unit, one residual block in the interior is as shown in formula (8):

[0088] x l+1 =h(x l )+F(x l ,W l ) (8)

[0089] Wherein, x l is the input of the lth layer, x l+1 is the output of the lth layer, h(x l ) is direct mapping, F(x l ,W l ) is the residual part, which is composed of two or three convolution layers, when x l and x l+1When the number of feature maps is equal, h(x) l )=x l And when x l With x l+1 When the number of feature maps is unequal, convolution can be used to increase or decrease dimensionality, h(x l As shown in formula (9):

[0090] h(x l ) = W l 'x l (9)

[0091] Among them, W l ' is a convolutional layer with a kernel size of 1×1.

[0092] The output information of the first one-dimensional convolutional network unit is: a one-dimensional feature of size Batch_Size×Window_Size×6n;

[0093] The output information of the first two-dimensional convolutional network unit is: two-dimensional features of size Batch_Size×Window_Size×3×2n;

[0094] The output information of the first 3D convolutional network unit is: a 3D feature of size Batch_Size×Window_Size×2×3×2n;

[0095] The behavioral information, after passing through c one-dimensional convolutional network units, outputs information of size Batch_Size × Window_. One-dimensional features;

[0096] The behavioral information, after passing through c two-dimensional convolutional network units, outputs information of size Batch_Size × Window_. Two-dimensional features;

[0097] The behavioral information, after passing through c 3D convolutional network units, outputs information of size Batch_Size × Window_. Three-dimensional features;

[0098] Step S44: After processing by the feature fusion unit, the three sizes are Batch_Size × Window_ Batch_Size×Window_ Batch_Size×Window_ The feature information is uniformly shaped into Batch_Size × Window_ The sum of the sizes is Batch_Size × Window_ Feature information;

[0099] Step S45: Set the number of attention heads of the multi-head attention unit as m, as shown in formula (10):

[0100]

[0101] Wherein, d k represents the length of K i , which is also equal to the value of the number of input attention feature map layers divided by the number of attention heads, Q i ,K i ,V i is the parameter corresponding to each attention head, which is obtained by multiplying the input sequence I by different weight matrices W, as shown in formula (11):

[0102]

[0103] Use head i to represent the output of each attention head, then after splicing the output of each attention head, multiply by the learned parameter W O to get the final result, as shown in formula (12):

[0104] MutiHead(Q,K,V)=Concat(head1,…,head m )W O (12)

[0105] The size of the information output after the multi-head attention unit is: Batch_Size×Window_

[0106] Step S46: output the recognition result through the output unit;

[0107] Through the expansion layer, the behavior information is converted from a vector to a scalar, and the size of the information output after the fully connected layer is: Batch_Size×Output_Length; the behavior information is classified and recognized by the Softmax classifier; the behavior information feature with the size of Batch_Size×Output_Length is input into the Softmax classifier, and the probability of each type of behavior is solved, and the behavior corresponding to the maximum probability is the final behavior recognition result of the MDC network module.

[0108] A computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the real-time behavior recognition method based on multi-source multi-modal data and multi-dimensional convolution model when executing the computer program.

[0109] A computer readable storage medium, having stored thereon a computer program, the computer program being executed by a processor to implement steps of a real-time behavior recognition method based on multi-source multi-modal data and a multi-dimensional convolution model.

[0110] The present application has the following advantages:

[0111] 1. Practicality: Real-time human behavior recognition has high requirements for transmission medium and accuracy, the present application uses a suitable transmission method to minimize the information transmission delay of the system; at the same time, it also has certain advantages in the accuracy of behavior recognition.

[0112] 2. Adaptability: For different application scenarios, by modifying the data remodeling unit parameters and convolution network unit structure, the universality of the input data is improved.

[0113] 3. High reliability: Compared with mainstream model algorithms, the use of multi-source multi-modal sensor data and filtering algorithms makes the present system have strong robustness, and the accuracy is further improved. BRIEF DESCRIPTION OF DRAWINGS

[0114] Figure 1 is a schematic diagram of the main module structure framework and connection relationship of the real-time behavior recognition system based on multi-source multi-modal data and multi-dimensional convolution model of the present application.

[0115] Figure 2 is a schematic diagram of the module composition and connection relationship of the real-time behavior recognition system based on multi-source multi-modal data and multi-dimensional convolution model of the present application.

[0116] Figure 3 is a flowchart of the real-time behavior recognition method based on multi-source multi-modal data and multi-dimensional convolution model of the present application.

[0117] Figure 4 is a schematic diagram of the working principle of the MDC network module of the present application.

[0118] Figure 5 is a schematic diagram of the network structure of the one-dimensional convolution network unit, two-dimensional convolution network unit and three-dimensional convolution network unit in the MDC network module of the present application. DETAILED DESCRIPTION

[0119] In order to make the purpose, technical scheme and advantages of the present application clearer and more apparent, the following combines embodiments and drawings of the specification Figures 1-3 The technical scheme of the present application is described clearly and completely, obviously, the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0120] Example 1

[0121] A real-time behavior recognition system based on multi-source multi-modal data and multi-dimensional convolution model, as shown in Figure 1 includes a behavior information physical layer, a behavior information access layer, a behavior information platform layer and a behavior information application layer connected in turn;

[0122] The behavior information physical layer is used for collecting behavior information of a user through a multi-source multi-modal wearable sensor device, and the behavior information includes X, Y and Z three-axis data of an acceleration sensor and X, Y and Z three-axis data of an angular velocity sensor;

[0123] The behavior information access layer is used for transmitting the collected behavior information to the behavior information access layer through a wireless network;

[0124] The behavior information platform layer is used for preprocessing the behavior information, specifically filtering, normalizing, multi-source multi-modal sensor data fusion and time series-based truncation in sequence, and inputting the truncated behavior data into a built behavior recognition model, and storing the output behavior recognition result and the original behavior information in a database server in real time; the multi-dimensional convolution model is the behavior recognition model;

[0125] Normalization refers to mean normalization processing of numerical data in the behavior information to avoid adverse effects of different dimensions and value ranges on calculation, that is, transforming values of all similar sensors to [-1, 1], thereby improving the accuracy and generalization ability of the behavior recognition model;

[0126] Multi-source multi-modal sensor data fusion refers to aligning and fusing behavior information carried by different sensors according to time stamps;

[0127] Time series-based truncation refers to segmenting continuous time series collected by a sensor through a sliding window method, and in order to ensure that a complete action data falls within a sliding window, the size and cutting frequency of the sliding window need to be determined according to the sensor sampling rate, for example, if the sensor sampling rate is 20 Hz and an action time is 2 s, then the size of the window should be at least 40 to contain a complete action data, and in order to ensure the integrity of the information, the sliding window moving step is generally half of the window size, that is, a window with a size of 40 is used to cut the data every second, and then the window slides back by 20, preparing for the next cutting;

[0128] The behavior information application layer is used for displaying the behavior recognition result output by the behavior recognition model.

[0129] The application proposes a relatively optimized system in four aspects of transmission technology, information processing, behavior recognition and behavior application, not only makes up for the deficiencies of poor real-time performance, inability to work in blind area and vulnerability to light in the current research field of behavior recognition, but also has certain improvement in the accuracy of behavior recognition, so that the system is more stable.

[0130] Embodiment 2

[0131] The real-time behavior recognition system based on multi-source multi-modal data and multi-dimensional convolution model according to Embodiment 1 is different in that:

[0132] As shown in Figure 2 , the behavior information physical layer includes a behavior information acquisition module, and the behavior information acquisition module includes a plurality of sensor modules; the sensor modules adopt IMU sensors to collect X, Y and Z three-axis data of an acceleration sensor and X, Y and Z three-axis data of an angular velocity sensor, thereby providing data support for behavior recognition.

[0133] The behavior information access layer includes a behavior information transmission module, and the behavior information transmission module adopts wireless network to transmit the behavior information collected by the behavior information acquisition module. The behavior information transmission module adopts any one of wireless communication technology, Bluetooth, ZigBee, fourth generation mobile communication technology, fifth generation mobile communication technology and long-distance radio to transmit the behavior information collected by the behavior information acquisition module. Transmission technology refers to a technology that fully utilizes the transmission capacity of different channels to form a complete transmission system, so that information can be reliably transmitted. With the development of wireless transmission technology, the convenience of wireless transmission is further magnified under the premise that there is no special requirement for the packet loss rate. At present, the mainstream wireless technologies mainly include WiFi, Bluetooth, ZigBee, 4G, 5G, LoRa and the like. Most of the wireless technologies are in a state of being unable to be compatible in terms of transmission distance and power consumption. The application can select a suitable transmission mode according to different scenes where the user is located, the portability of the wearable device and the requirements of real-time behavior recognition.

[0134] The behavior information platform layer includes a server module, a behavior information preprocessing module and an MDC network module.

[0135] The behavior information preprocessing module is used to preprocess behavior information, thereby improving its reliability. Compared with some systems that directly identify behavior from raw behavior information, this invention significantly improves system stability and recognition accuracy after incorporating behavior information preprocessing. The behavior information preprocessing module includes a behavior information denoising unit, a behavior information normalization unit, a behavior information fusion unit, and a behavior information sliding window unit. The behavior information transmission module sends the behavior information to the behavior information denoising unit, which denoises the behavior information using wavelet thresholding. The denoising unit then processes the behavior information using mean normalization. The behavior information fusion unit aligns and fuses the normalized multi-source, multi-modal sensor behavior information according to timestamps, and then inputs it to the behavior information sliding window unit. The behavior information is then truncated by adjusting the size, cutting frequency, and step size of the sliding window.

[0136] The MDC network module includes a data reshaping unit, a one-dimensional convolutional network unit, a two-dimensional convolutional network unit, a three-dimensional convolutional network unit, a feature fusion unit, a multi-head attention unit, and an output unit. The data reshaping unit adjusts the dimensions of the behavioral information output from the behavioral information preprocessing module, segmenting it according to sensor type and wearing position, so that the original one-dimensional time series carries spatial information, reshaping the original one-dimensional behavioral data into two dimensions: two-dimensional and three-dimensional behavioral data. The one-dimensional, two-dimensional, and three-dimensional behavioral data are then input into the one-dimensional, two-dimensional, and three-dimensional convolutional network units, respectively. The one-dimensional convolutional network unit includes three one-dimensional convolutional kernels, which are well used to extract the temporal characteristics of behavioral information during the convolution process. The two-dimensional convolutional network unit includes three... Two-dimensional convolutional kernels can focus on the relationships between data from different sensors during convolution, thus fully extracting the spatial characteristics of behavioral information. Three-dimensional convolutional network units, comprising three three-dimensional convolutional kernels, can focus on the relationships between sensor data from different wearing positions, thus extracting the deep spatial characteristics of behavioral information. The behavioral information output by convolutional network units of different dimensions has different dimensions. The extracted temporal, spatial, and deep spatial characteristics of the behavioral information are fused by a feature fusion unit and then input into a multi-head attention unit to analyze the relationship between the time series and space, obtaining global dependencies. The resulting data is input to the output unit, sequentially passing through an unfolded layer, a fully connected layer, and a Softmax classifier to process all feature data and identify the current behavior.

[0137] The main work of the MDC network module is to make identification according to behavior information, and the proposal of artificial intelligence and machine learning makes people realize the power of artificial intelligence. Compared with traditional machine learning, deep learning has made obvious progress in recognition rate. However, the traditional machine learning model and the basic deep learning model mainly focus on the shallow features or incomplete features of the input information. The MDC network used in the application not only pays attention to the time characteristics of the behavior information, but also adds the spatial relationship of the behavior information characteristics, thereby improving the accuracy of behavior recognition.

[0138] The server module includes a database server unit for storing original behavior information and behavior recognition results.

[0139] The behavior information application layer includes a behavior and location information display module connected with the output unit in the MDC network module, for displaying the behavior recognition results output by the output unit. In the application aspect, a dangerous behavior warning module can be additionally attached, and when a dangerous behavior is identified, an alarm is given.

[0140] Embodiment 3

[0141] A real-time behavior recognition method based on multi-source multi-modal data and multi-dimensional convolution model is realized through the real-time behavior recognition system based on multi-source multi-modal data and multi-dimensional convolution model described in embodiment 2. Taking the behavior recognition of fighting as an example, special personnel may be slightly different from ordinary people in terms of psychology and physiology, and may have radical behavior in dealing with some problems. In order to more accurately and reliably recognize the overactive behavior and prevent serious influence and harm, the behavior information of the special personnel is acquired through multiple sensors worn by the special personnel, then the information is preprocessed to improve the reliability of the information, and finally the trained behavior recognition model is used for real-time behavior recognition and monitoring and early warning. Managers can set different parameters for real-time behavior recognition according to different scenes and different special personnel. Figure 3 As shown, the method comprises the following steps:

[0142] Step S1: behavior information acquisition

[0143] The sensors are worn at different positions of the user to collect the behavior information of the user, and the behavior information includes X, Y and Z three-axis data of the acceleration sensor and X, Y and Z three-axis data of the angular velocity sensor.

[0144] The sensors are selected according to practical scenes, user requirements and related parameters of the sensors, and the related parameters of the sensors include a sampling frequency, a service life, precision, power consumption and the like. For example, for a user who needs to be monitored, a sensor with high sampling frequency and high precision can be selected to perceive behavior information, and for a general user, a sensor with a sampling frequency and precision meeting basic requirements can be selected to perceive behavior information.

[0145] Step S2: behavior information transmission

[0146] The collected behavior information is transmitted to a local server or a cloud server; according to different scenes where the user is located, the portability of the wearable device and the requirements of real-time behavior recognition, a suitable transmission mode such as WiFi, LoRa, Bluetooth and the like can be selected. Different transmission modes are different in terms of power consumption, transmission distance, transmission speed and the like, and therefore, the method can be applied to different application scenes.

[0147] Step S3: behavior information preprocessing

[0148] The behavior information is preprocessed, specifically, filtering, normalization, multi-source multi-modal sensor data fusion and time series-based interception are sequentially performed, including the following steps.

[0149] Step S31: behavior information denoising

[0150] The behavior information is denoised by a wavelet threshold method; according to the data type, appropriate wavelet transform parameters and threshold functions can be selected to filter part of the sensor noise signals in the collection process, so that the behavior data becomes smoother, which is conducive to accelerating the convergence of the network model and improving the classification accuracy.

[0151] The wavelet threshold method denoising processing mode has the characteristics of simple implementation and small amount of calculation. Specifically, it is that:

[0152] Suppose the actual measurement signal is f(t) = s(t) + e(t), t = 1, 2,..., N, s(t) is the original signal, f(t) is the noise signal, e(t) is the Gaussian white noise, e(t) ~ N(0, σ 2 ), σ is the noise intensity, the denoising process is to remove the noise e(t) from the signal f(t) to obtain the best approximation of the original signal s(t);

[0153] Generally, the threshold denoising process of a one-dimensional signal is divided into three steps:

[0154] First, it is discretely sampled to obtain an N' point discrete signal f(x), x = 0, 1, 2,..., N'-1, and the wavelet transform coefficient is as shown in formula (2):

[0155]

[0156] wherein W f (j,k) is the wavelet coefficient, ψ(2 -j x-k) is the scaling function, j is the scale parameter, k is the unit number of the scaling function translation, in practical application, it is more complex to directly use formula (2) for calculation, and ψ(t) generally has no analytical expression. However, through double-scale equations (3) and (4), a recursive implementation method of formula (2) is obtained.

[0157] S f (j+1,k)=S f (j,k)*h(j,k) (3)

[0158] W f (j+1,k)=S f (j,k)*g(j,k) (4)

[0159] wherein the symbol "*" represents convolution, h and g represent low-pass and high-pass filters respectively, S f (0,k) represents the original signal f(k), S f (j,k) represents the approximation coefficient at the j scale, and then the wavelet transform reconstruction formula is shown in formula (5):

[0160] S f (j-1,k)=S f (j,k)*h(j,k)+W f (j,k)*g(j,k) (5)

[0161] Secondly, a general threshold rule is used to determine the threshold, and the threshold algorithm formula is shown in formula (6):

[0162]

[0163] wherein σ=MAD / 0.6745, MAD is the median value of the absolute value of the first layer wavelet decomposition coefficient, 0.6745 is the adjustment coefficient of the standard deviation of the Gaussian noise, and L is the size or length of the signal;

[0164] Then, the wavelet transform coefficients of the measurement signal are subjected to nonlinear threshold processing, for each layer of high-frequency coefficients from the first layer to the Vth layer, a soft threshold function is used for processing, the absolute value of the signal is compared with the threshold value, the points less than the threshold value are set to zero, and the points greater than or equal to the threshold value are shrunk to zero, becoming the difference between the point value and the threshold value, and the low-frequency coefficients of each layer are not processed; the soft threshold function is shown in formula (7):

[0165]

[0166] wherein W j,k is Wf (j,k) is a simplified form where T is a threshold parameter greater than zero.

[0167] Finally, based on the low-frequency coefficients of the Nth layer of wavelet decomposition and the high-frequency coefficients of the 1st to Vth layers after quantization, wavelet reconstruction of the signal is performed according to Equation (4) to obtain the denoised signal.

[0168] Step S32: Normalization of behavioral information

[0169] Mean normalization is performed to bring features of different dimensions to the same numerical order of magnitude; this not only makes the network model more accurate, but also speeds up the convergence of the learning algorithm.

[0170] Step S33: Multi-source multimodal sensor data fusion

[0171] The behavioral information carried by sensors worn in different locations is aligned according to timestamps; in case of sensor data loss, missing values ​​are automatically filled with "0"; creating conditions for the subsequent extraction of spatial information;

[0172] Step S34: Time series-based extraction

[0173] Referring to user-preset parameters, including the size of the sliding window, the cutting frequency, and the step size of the sliding window, the behavior information processed in step S33 is processed by sliding window, so that the behavior information is input into the MDC network module in the form of information blocks.

[0174] Step S4: Build and train the behavior recognition model

[0175] The behavior recognition model, or MDC network module, includes a data reshaping unit, a one-dimensional convolutional network unit, a two-dimensional convolutional network unit, a three-dimensional convolutional network unit, a feature fusion unit, a multi-head attention unit, and an output unit.

[0176] The parameters set by the user include: the shape of the input data, the shape of the reshaped data, the number of convolutional network units, etc. The learning rate is set to 0.0005, and the cross-entropy loss function is used. Through iterative training, the loss function value is continuously reduced, and the parameters of the behavior recognition model are further optimized. Finally, the trained behavior recognition model is obtained. The loss function formula is shown in Equation (1):

[0177]

[0178] Where A is the number of samples, M is the number of categories, and y ij The sign function is p (0 or 1), which takes the value 1 if the true class of sample i is equal to j, and 0 otherwise. ij Let be the predicted probability that observed sample i belongs to category j.

[0179] Setting the shape of the input data, the one-dimensional convolutional network unit convolution kernel size, the two-dimensional convolutional network unit convolution kernel size, the three-dimensional convolutional network unit convolution kernel size, the number of one-dimensional convolutional network units, the number of two-dimensional convolutional network units, the number of three-dimensional convolutional network units, and the number of attention heads of the multi-head attention unit;

[0180] Setting the two-dimensional data reshaping parameters x, y, z and the three-dimensional data reshaping parameters r, s, t, u of the data reshaping unit; x, y, z are the length, width, and channel number of the reshaped two-dimensional data, respectively; r, s, t, u are the length, width, height, and channel number of the reshaped three-dimensional data, respectively;

[0181] Setting Batch_Size and Window_Size; Batch_Size refers to the number of behavior samples in each batch in the MDC network module, and Window_Size refers to the length of the sample data;

[0182] Setting the convolution kernel size of the first convolution layer in the i-th one-dimensional convolutional network unit to 1 and the number to o1 i , the convolution kernel size of the second convolution layer to 3 and the number to p1 i , the convolution kernel size of the third convolution layer to 1 and the number to q1 i , and the padding mode of the three convolution layers to "SAME";

[0183] Setting the convolution kernel size of the first convolution layer in the i-th two-dimensional convolutional network unit to 1x1 and the number to o2 i , the convolution kernel size of the second convolution layer to 3x3 and the number to p2 i , the convolution kernel size of the third convolution layer to 1x1 and the number to q2 i , and the padding mode of the three convolution layers to "SAME";

[0184] Setting the convolution kernel size of the first convolution layer in the i-th three-dimensional convolutional network unit to 1x1x1 and the number to o3 i , the convolution kernel size of the second convolution layer to 3x3x3 and the number to p3 i , the convolution kernel size of the third convolution layer to 1x1x1 and the number to q3 i , and the padding mode of the three convolution layers to "SAME";

[0185] Setting the number of one-dimensional convolutional network units, two-dimensional convolutional network units, and three-dimensional convolutional network units to c;

[0186] Setting the reshaping parameters of the feature fusion unit

[0187] Setting the full connection output length, i.e., the number of classifications, in the output unit to Output_Length;

[0188] As Figure 4 shown, comprising the following steps:

[0189] Step S41: Assuming that the input data is composed of sensor data fusion of n inertial measurement units, the behavior information with a size of Batch_Size x Window_Size x 6n, Window_Size is the length of one-dimensional data, and 6n is the number of data channels;

[0190] Step S42: After the behavior information with a size of Batch_Size x Window_Size x 6n passes through the data remodeling unit, two forms of Batch_Size x Window_Size x 3 x 2n and Batch_Size x Window_Size x 2 x 3 x n are generated, 3 x 2n represents that 2n sensor modules each carry 3-axis behavior data, and the latter is divided on the basis of the former, and 2n sensor units are divided into n accelerometers and n gyroscopes, and 2 x 3 x n represents that n inertial measurement units of sensors each carry an accelerometer and a gyroscope, and the data of each sensor is 3-axis data;

[0191] Step S43: The behavior information with a size of Batch_Size x Window_Size x 6n, Batch_Size x Window_Size x 3 x 2n, and Batch_Size x Window_Size x 2 x 3 x n is respectively input into a one-dimensional convolution network unit, a two-dimensional convolution network unit, and a three-dimensional convolution network unit, the one-dimensional convolution network unit, the two-dimensional convolution network unit, and the three-dimensional convolution network unit all adopt a residual structure, and a residual block includes a direct mapping part and a residual part, in the two-dimensional convolution network unit, one residual block in the interior is shown in formula (8):

[0192] x l+1 =h(x l )+F(x l ,W l ) (8)

[0193] Wherein, x l is the input of the lth layer, x l+1 is the output of the lth layer, h(x l ) is a direct mapping, F(x l ,W l ) is a residual part, which is composed of two or three convolution layers, when the feature map number of x l is equal to that of x l+1 , h(x l ) = x l , and when the feature map number of x l is not equal to that of x l+1The number of feature maps is different, and the dimension is increased or decreased using convolution, h(x l ) is shown in formula (9):

[0194] h(x l )=W l 'x l (9)

[0195] Where W l ' is a 1x1 convolution layer with a convolution kernel size of 1x1.

[0196] The output information of the first one-dimensional convolution network unit is: one-dimensional feature with a size of Batch_SizexWindow_Sizex6n;

[0197] The output information of the first two-dimensional convolution network unit is: two-dimensional feature with a size of Batch_SizexWindow_Sizex3x2n;

[0198] The output information of the first three-dimensional convolution network unit is: three-dimensional feature with a size of Batch_SizexWindow_Sizex2x3x2n;

[0199] The output information of the behavior information after passing through c one-dimensional convolution network units is: one-dimensional feature with a size of Batch_SizexWindow_ ;

[0200] The output information of the behavior information after passing through c two-dimensional convolution network units is: two-dimensional feature with a size of Batch_SizexWindow_ ;

[0201] The output information of the behavior information after passing through c three-dimensional convolution network units is: three-dimensional feature with a size of Batch_SizexWindow_ ; The structure of the one-dimensional convolution network unit, the two-dimensional convolution network unit, and the three-dimensional convolution network unit is shown in Figure 5 .

[0202] Step S44: After processing by the feature fusion unit, the three feature information with sizes of Batch_SizexWindow_ , Batch_SizexWindow_ , and Batch_SizexWindow_ are reshaped to Batch_SizexWindow_ , and the feature information with a size of Batch_SizexWindow_ is output after addition;

[0203] Step S45: Set the number of attention heads of the multi-head attention unit as m, as shown in formula (10):

[0204]

[0205] wherein d k represents the length of K i , which is also equal to the value of the number of input attention feature map layers divided by the number of attention heads, Q i ,K i ,V i are parameters corresponding to each attention head, which are obtained by multiplying the input sequence I by different weight matrices W, as shown in formula (11):

[0206]

[0207] Use head i to represent the output of each attention head, then multiply the output of each attention head by the learned parameter W O after splicing, to obtain the final result, as shown in formula (12):

[0208] MutiHead(Q,K,V)=Concat(head1,…,head m )W O (12)

[0209] The size of the information output after the multi-head attention unit is: Batch_Size×Window_

[0210] Step S46: Output the recognition result through the output unit;

[0211] Through the expansion layer, the behavior information is converted from a vector to a scalar, and the size of the information output after the fully connected layer is: Batch_Size×Output_Length; the behavior information is classified and recognized by the Softmax classifier; the behavior information feature with the size of Batch_Size×Output_Length is input into the Softmax classifier, and the probability of each behavior is solved, and the behavior corresponding to the maximum probability is the final behavior recognition result of the MDC network module.

[0212] Step S5: Real-time recognition of behavior information

[0213] The real-time collected and preprocessed behavior information is input into the trained behavior recognition model for real-time recognition of the current behavior, and the classification result corresponding to the behavior information is output;

[0214] Step S6: Display of behavior information

[0215] The behavior recognition result is displayed by the behavior information display module.

[0216] Embodiment 4

[0217] A computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the real-time behavior recognition method based on multi-source multi-modal data and multi-dimensional convolution model in embodiment 3 when executing the computer program.

[0218] Embodiment 5

[0219] A computer readable storage medium, having a computer program stored thereon, the computer program, when executed by a processor, implements the steps of the real-time behavior recognition method based on multi-source multi-modal data and multi-dimensional convolution model in embodiment 3.

Claims

1. A real-time behavior recognition system based on multi-source multimodal data and a multidimensional convolutional model, characterized in that, This includes the physical layer of behavioral information, the access layer of behavioral information, the platform layer of behavioral information, and the application layer of behavioral information, which are connected in sequence. The physical layer of behavioral information is used to: collect user behavioral information, which includes X, Y, and Z axis data from an accelerometer and X, Y, and Z axis data from an angular velocity sensor; The behavior information access layer is used to: transmit the collected behavior information to the behavior information access layer; The behavior information platform layer is used to: preprocess behavior information, specifically by: sequentially performing filtering, normalization, multi-source multimodal sensor data fusion and time-series-based truncation, and inputting the truncation behavior data into the constructed behavior recognition model, and storing the output behavior recognition results and the original behavior information in real time; the multidimensional convolution model is the behavior recognition model; Normalization refers to the mean normalization process performed on numerical data in behavioral information, that is, transforming the values ​​of all similar sensors to the range of [-1, 1]. Multi-source, multi-modal sensor data fusion refers to the process of merging behavioral information carried by different sensors after aligning them according to timestamps. Time series-based segmentation refers to segmenting continuous time series acquired by sensors using a sliding window method to ensure that a complete set of motion data falls within a single sliding window. The behavior information application layer is used to: display the behavior recognition results output by the behavior recognition model; The behavioral information platform layer includes a server module, a behavioral information preprocessing module, and an MDC network module; The behavior information preprocessing module is used to preprocess the behavior information. The behavior information preprocessing module includes a behavior information denoising unit, a behavior information normalization unit, a behavior information fusion unit, and a behavior information sliding window unit. The behavior information transmission module sends the behavior information to the behavior information denoising unit. After the behavior information denoising unit denoises the behavior information using the wavelet thresholding method, it is input into the behavior information normalization unit for processing using the mean normalization method. The behavior information fusion unit aligns and fuses the normalized multi-source multimodal sensor behavior information according to the timestamps, and then inputs it into the behavior information sliding window unit. The behavior information is truncated by adjusting the size of the sliding window, the cutting frequency, and the step size of the sliding window. The MDC network module includes a data reshaping unit, a one-dimensional convolutional network unit, a two-dimensional convolutional network unit, a three-dimensional convolutional network unit, a feature fusion unit, a multi-head attention unit, and an output unit. The data reshaping unit adjusts the dimensions of the behavioral information output by the behavioral information preprocessing module, segmenting it according to sensor type and wearing position, so that the original one-dimensional time series carries spatial information, reshaping the original one-dimensional behavioral data into two-dimensional and three-dimensional behavioral data. The one-dimensional, two-dimensional, and three-dimensional behavioral data are then input into the one-dimensional convolutional network unit, the two-dimensional convolutional network unit, and the three-dimensional convolutional network unit, respectively. The one-dimensional convolutional network unit includes three one-dimensional convolutional networks. The convolutional kernel is used to extract the temporal characteristics of behavioral information; the two-dimensional convolutional network unit includes three two-dimensional convolutional kernels to extract the spatial characteristics of behavioral information; the three-dimensional convolutional network unit includes three three-dimensional convolutional kernels to extract the deep spatial characteristics of behavioral information; the extracted temporal characteristics, spatial characteristics, and deep spatial characteristics of behavioral information are fused by the feature fusion unit and then input into the multi-head attention unit to analyze the relationship between time series and space, obtain global dependencies, and the resulting data is input into the output unit, which sequentially passes through the unfolded layer, the fully connected layer, and the Softmax classifier to perform calculations on all feature data and identify the current behavior; The server module includes a database server unit for storing raw behavioral information and behavioral recognition results.

2. The real-time behavior recognition system based on multi-source multimodal data and a multidimensional convolutional model according to claim 1, characterized in that, The physical layer of behavioral information includes a behavioral information acquisition module, which includes multiple sensor modules; the sensor modules use IMU sensors to acquire triaxial acceleration data and triaxial angular velocity data. The behavior information access layer includes a behavior information transmission module, which uses a wireless network to transmit the behavior information collected by the behavior information acquisition module. The behavior information transmission module uses any one of the following transmission technologies—WiFi, Bluetooth, ZigBee, 4G, 5G, and LoRa—to transmit the behavior information collected by the behavior information acquisition module.

3. The real-time behavior recognition system based on multi-source multimodal data and a multidimensional convolutional model according to claim 1, characterized in that, The behavior information application layer includes a behavior and location information display module; the behavior and location information display module is connected to the output unit in the MDC network module and is used to display the behavior recognition results output by the output unit.

4. A real-time behavior recognition method based on multi-source multimodal data and a multidimensional convolutional model, implemented by the real-time behavior recognition system based on multi-source multimodal data and a multidimensional convolutional model as described in any one of claims 1-3, characterized in that, Includes the following steps: Step S1: Behavioral Information Collection Collect user behavior information, including triaxial acceleration data and triaxial angular velocity data generated by human behavior; Step S2: Behavioral Information Transmission The collected behavioral information is transmitted to a local server or a cloud server; Step S3: Behavioral Information Preprocessing Preprocessing behavioral information specifically refers to: sequentially performing filtering, normalization, multi-source multimodal sensor data fusion, and time-series-based data extraction; Step S4: Build and train the behavior recognition model The behavior recognition model, namely the MDC network module, includes a data reshaping unit, a one-dimensional convolutional network unit, a two-dimensional convolutional network unit, a three-dimensional convolutional network unit, a feature fusion unit, a multi-head attention unit, and an output unit; The parameters set by the user include: the shape of the input data, the shape of the reshaped data, and the number of convolutional network units. Through iterative training, the parameters of the behavior recognition model are continuously optimized, and finally the trained behavior recognition model is obtained. Step S5: Real-time recognition of behavioral information Real-time collected and preprocessed behavioral information is input into a pre-trained behavior recognition model to perform real-time recognition of the current behavior and output the classification result corresponding to the behavioral information. Step S6: Display of Behavioral Information The behavior recognition results are displayed through the behavior information display module; In step S4, the learning rate is set to 0.0005, and the cross-entropy loss function is used. During the iterative training process, the loss function value continuously decreases, and the parameters of the behavior recognition model are further optimized, finally obtaining the trained behavior recognition model. The loss function formula is shown in formula (1): Where A is the number of samples, M is the number of categories, and y ij The sign function is p, which takes the value 1 if the true class of sample i is equal to j, and 0 otherwise. ij The predicted probability that observed sample i belongs to category j; Step S1 refers to: wearing the sensor at different positions on the user to obtain sensor data, namely acceleration and angular velocity data.

5. The real-time behavior recognition method based on multi-source multimodal data and a multidimensional convolutional model according to claim 4, characterized in that, Step S3 includes the following steps: Step S31: Denoising of behavioral information Denoising of behavioral information is performed using wavelet thresholding. Step S32: Normalization of behavioral information Mean normalization is performed to bring features of different dimensions to the same numerical order of magnitude. Step S33: Multi-source multimodal sensor data fusion Align the behavioral information carried by sensors worn in different locations according to timestamps; Step S34: Time series-based extraction Referring to user-preset parameters, including the size of the sliding window, the cutting frequency, and the step size of the sliding window, the behavior information processed in step S33 is processed by sliding window, so that the behavior information is input into the MDC network module in the form of information blocks.

6. The real-time behavior recognition method based on multi-source multimodal data and a multidimensional convolutional model according to claim 4, characterized in that, In step S31, the behavioral information is denoised using the wavelet thresholding method, specifically: Assume the actual measured signal is: f(t) = s(t) + e(t), t = 1, 2, ..., N, where s(t) is the original signal, f(t) is the noisy signal, e(t) is Gaussian white noise, and e(t) ~ N(0, σ 2 ), where σ is the noise intensity, and the denoising process is to remove the noise e(t) from the signal f(t) to obtain the best approximation of the original signal s(t); First, discrete sampling is performed to obtain N'-point discrete signal f(x), x=0,1,2,…,N'-1, and its wavelet transform coefficients are shown in formula (2): Among them, W f (j,k) are wavelet coefficients, ψ(2 -j xk) is the scaling function, j is the scaling parameter, and k is the number of units the scaling function is shifted. Through the double scaling equations (3) and (4), the recursive implementation method of equation (2) is obtained: S f (j+1,k)=S f (j,k)*h(j,k)(3) W f (j+1,k)=S f (j,k)*g(j,k)(4) where the symbol "*" represents convolution, h and g represent low-pass and high-pass filters respectively, and S f (0,k) represents the original signal f(k), S f (j,k) represents the approximation coefficient at scale j, and the wavelet transform reconstruction formula is shown in formula (5): S f (j-1,k)=S f (j,k)*h(j,k)+W f (j,k)*g(j,k)(5) Secondly, the threshold is determined using a general threshold rule, and the threshold algorithm formula is shown in formula (6): Where σ = MAD / 0.6745, MAD is the median value of the absolute values ​​of the first-layer wavelet decomposition coefficients, 0.6745 is the adjustment coefficient of the standard deviation of Gaussian noise, and L is the size or length of the signal; Then, nonlinear thresholding is applied to the wavelet transform coefficients of the measured signal. For the high-frequency coefficients of each layer from layer 1 to layer V, a soft thresholding function is used. The absolute value of the signal is compared with the threshold. Points less than the threshold are set to zero, and points greater than or equal to the threshold are shrunk to zero, becoming the difference between the value of that point and the threshold. The low-frequency coefficients of each layer are not processed. The soft thresholding function is shown in formula (7): Among them, W j,k For W f (j,k) is a simplified form where T is a threshold parameter greater than zero; Finally, based on the low-frequency coefficients of the Nth layer of wavelet decomposition and the high-frequency coefficients of the 1st to Vth layers after quantization, wavelet reconstruction of the signal is performed according to Equation (4) to obtain the denoised signal.

7. The real-time behavior recognition method based on multi-source multimodal data and a multidimensional convolutional model according to claim 4, characterized in that, In step S4, the shape of the input data, the kernel size of the one-dimensional convolutional network unit, the kernel size of the two-dimensional convolutional network unit, the kernel size of the three-dimensional convolutional network unit, the number of one-dimensional convolutional network units, the number of two-dimensional convolutional network units, the number of three-dimensional convolutional network units, and the number of attention heads of the multi-head attention unit are set. Set the two-dimensional data reshaping parameters x, y, z, and the three-dimensional data reshaping parameters r, s, t, u for the data reshaping unit; x, y, z are the length, width, and number of channels of the reshaped two-dimensional data, respectively; r, s, t, u are the length, width, height, and number of channels of the reshaped three-dimensional data, respectively. Set Batch_Size and Window_Size. Batch_Size refers to the number of behavioral samples in each batch in the MDC network module, and Window_Size refers to the length of the sample data. The kernel size of the first convolutional layer in the i-th one-dimensional convolutional network unit is set to 1, and the number of kernels is o1. i The second convolutional layer has a kernel size of 3 and a number of p1 kernels. i The third convolutional layer has a kernel size of 1 and a number of kernels of size q1. i All three convolutional layers are padded with "SAME"; The kernel size of the first convolutional layer in the i-th two-dimensional convolutional network unit is set to 1×1, and the number of kernels is o2. i The second convolutional layer has a kernel size of 3×3 and a number of p2 kernels. i The third convolutional layer has a kernel size of 1×1 and a number of q² kernels. i All three convolutional layers are padded with "SAME"; The kernel size of the first convolutional layer in the i-th 3D convolutional network unit is set to 1×1×1, and the number of kernels is o3. i The second convolutional layer has a kernel size of 3×3×3 and a number of p3 kernels. i The third convolutional layer has a kernel size of 1×1×1 and a kernel size of q³. i All three convolutional layers are padded with "SAME"; Set the number of one-dimensional convolutional network units, two-dimensional convolutional network units, and three-dimensional convolutional network units to c; Set feature fusion unit shaping parameters Set the fully connected output length, i.e. the number of categories, in the output unit to Output_Length; Includes the following steps: Step S41: Assume that the input data is composed of sensor data fusion from n inertial measurement units, and the size is Batch_Size×Window_Size×6n behavioral information, where Window_Size is the one-dimensional data length and 6n is the number of data channels; Step S42: After the behavior information of size Batch_Size×Window_Size×6n passes through the data reshaping unit, it is reshaped into two forms: Batch_Size×Window_Size×3×2n and Batch_Size×Window_Size×2×3×n. 3×2n represents: 2n sensor modules, each sensor module carries 3-axis behavior data, and the 2n sensor units are divided into two categories: n accelerometers and n gyroscopes. 2×3×n represents: the sensors of n inertial measurement units, each sensor has an accelerometer and a gyroscope, and the data of each sensor is 3-axis data. Step S43: Input the behavioral information of sizes Batch_Size×Window_Size×6n, Batch_Size×Window_Size×3×2n, and Batch_Size×Window_Size×2×3×n into the one-dimensional convolutional network unit, two-dimensional convolutional network unit, and three-dimensional convolutional network unit, respectively. The one-dimensional convolutional network unit, two-dimensional convolutional network unit, and three-dimensional convolutional network unit all adopt residual structure. The residual block includes a direct mapping part and a residual part. In the two-dimensional convolutional network unit, one of its internal residual blocks is shown in formula (8): x l+1 =h(x l )+F(x l W l (8) Where, x l It is the input of the l-th layer, x l+1 It is the output of the l-th layer, h(x) l F(x) is a direct mapping. l W l ) is the residual part, consisting of two or three convolutional layers, when x l With x l+1 When the number of feature maps is equal, h(x) l )=x l And when x l With x l+1 When the number of feature maps is unequal, convolution can be used to increase or decrease dimensionality, h(x l As shown in formula (9): h(x l )=W l 'x l (9) Among them, W l ' is a convolutional layer with a kernel size of 1×1; The output information of the first one-dimensional convolutional network unit is: a one-dimensional feature of size Batch_Size×Window_Size×6n; The output information of the first two-dimensional convolutional network unit is: two-dimensional features of size Batch_Size×Window_Size×3×2n; The output information of the first 3D convolutional network unit is: a 3D feature of size Batch_Size×Window_Size×2×3×2n; The behavioral information output after passing through c one-dimensional convolutional network units is of size c. One-dimensional features; The behavioral information, after passing through c two-dimensional convolutional network units, outputs information of size: Two-dimensional features; The behavioral information, after passing through c 3D convolutional network units, outputs information of size: Three-dimensional features; Step S44: After processing by the feature fusion unit, the three sizes are respectively The feature information is uniformly shaped into The sum of the sizes will result in an output size of 1. Feature information; Step S45: Set the number of attention heads in the multi-head attention unit to m, as shown in formula (10): Where, d k Representing K i The length of Q is also equal to the number of input attention feature layers divided by the number of attention heads. i ,K i V i The parameters corresponding to each attention head are obtained by multiplying the input sequence I by different weight matrices W, as shown in formula (11): Use head i Let W represent the output of each attention head. Then, after concatenating the outputs of each attention head, multiply by the learned parameter W. O The final result is shown in formula (12): MutiHead(Q,K,V)=Concat(head1,…,head m )W O (12) The size of the information output after passing through the multi-head attention unit is: Step S46: Output the recognition result through the output unit; The behavior information is transformed from a vector to a scalar through the unfolding layer. After passing through the fully connected layer, the output information size is Batch_Size × Output_Length. The behavior information is classified and identified by the Softmax classifier. The behavior information features with an information size of Batch_Size × Output_Length are input into the Softmax classifier to solve for the probability of each behavior. The behavior corresponding to the highest probability is the final behavior recognition result of the MDC network module.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the real-time behavior recognition method based on multi-source multimodal data and multidimensional convolutional model as described in any one of claims 4-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the real-time behavior recognition method based on multi-source multimodal data and multidimensional convolutional model as described in any one of claims 4-7.

Images